gonymizer
gonymizer copied to clipboard
smithoss
gonymizer "process" ends with panic: runtime error: index out of range
Hi,
I am trying to use the docker image from smithoss/gonymizer and I am having partial success. Is the number of rows possibly capped at 3.000.000?
Here is my docker command:
docker build --no-cache --rm gonymizer
The Dockerfile:
#FROM junkert/gonymizer
FROM smithoss/gonymizer
ENV PG_SRC_HOST=
ENV PG_SRC_USER=
ENV PG_SRC_PSWD=
ENV PG_SRC_PORT=
ENV PG_SRC_DBNAME=grossolini_as
ENV PG_SRC_SCHEMA=as
ENV FILE_PREFIX=$PG_SRC_DBNAME
ENV SKEL_FILE=$FILE_PREFIX.skeleton.json
ENV TMP_FILE=$FILE_PREFIX.tmp.sql
ENV DUMP_FILE=$FILE_PREFIX.dump.sql
ENV DBG_LVL=DEBUG
RUN pg_dump --version && pg_restore --version && gonymizer version
RUN apk update
RUN apk add jq
RUN touch $SKEL_FILE && rm $SKEL_FILE
#RUN gonymizer map --map-file="$FILE_PREFIX" -S -H "$PG_SRC_HOST" -p "$PG_SRC_PSWD" -U "$PG_SRC_USER" -P "$PG_SRC_PORT" -d "$PG_SRC_DBNAME" --schema "$PG_SRC_SCHEMA" -L "$DBG_LVL"
RUN echo '{"DBName":"grossolini_as","SchemaPrefix":"","Seed":0,"ColumnMaps":[{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_login","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":16,"IsNullable":true,"Processors":[{"Name":"AlphaNumericScrambler","Max":0,"Min":0,"Variance":0,"Comment":""}]},{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_password","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":17,"IsNullable":true,"Processors":[{"Name":"ScrubString","Max":0,"Min":0,"Variance":0,"Comment":""}]}]}' | jq > $SKEL_FILE
RUN touch $TMP_FILE && rm $TMP_FILE
RUN gonymizer dump --dump-file="$TMP_FILE" -S -H "$PG_SRC_HOST" -p "$PG_SRC_PSWD" -U "$PG_SRC_USER" -P "$PG_SRC_PORT" -d "$PG_SRC_DBNAME" --schema "$PG_SRC_SCHEMA" -L "$DBG_LVL"
RUN gonymizer process --map-file="$SKEL_FILE" --dump-file="$TMP_FILE" --processed-file="$DUMP_FILE" --generate-seed -L "$DBG_LVL"
The output with the smithoss/gonymizer image (starting after all the ENV's):
Step 25/32 : RUN pg_dump --version && pg_restore --version && gonymizer version
---> Running in acc22444b47e
pg_dump (PostgreSQL) 11.3
pg_restore (PostgreSQL) 11.3
gonymizer (v1.1.1, build 5, build date:2019-05-06 23:50:02 +0000 UTC)
Go (runtime:go1.12.6) (GOMAXPROCS:2) (NumCPUs:2)
Removing intermediate container acc22444b47e
---> 30c431b3bc6c
Step 26/32 : RUN apk update
---> Running in 524cd4b2c078
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
v3.10.5-83-g4b863b300a [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
v3.10.5-73-g9ff6848e18 [http://dl-cdn.alpinelinux.org/alpine/v3.10/community]
OK: 10365 distinct packages available
Removing intermediate container 524cd4b2c078
---> 2dfe825e40b6
Step 27/32 : RUN apk add jq
---> Running in 6f1fdc068f01
(1/2) Installing oniguruma (6.9.4-r0)
(2/2) Installing jq (1.6-r0)
Executing busybox-1.30.1-r2.trigger
OK: 37 MiB in 32 packages
Removing intermediate container 6f1fdc068f01
---> dc8d01962a14
Step 28/32 : RUN touch $SKEL_FILE && rm $SKEL_FILE
---> Running in a3525973e8d8
Removing intermediate container a3525973e8d8
---> bf1226931dee
Step 29/32 : RUN echo '{"DBName":"grossolini_as","SchemaPrefix":"","Seed":0,"ColumnMaps":[{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_login","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":16,"IsNullable":true,"Processors":[{"Name":"AlphaNumericScrambler","Max":0,"Min":0,"Variance":0,"Comment":""}]},{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_password","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":17,"IsNullable":true,"Processors":[{"Name":"ScrubString","Max":0,"Min":0,"Variance":0,"Comment":""}]}]}' | jq > $SKEL_FILE
---> Running in d4127754615f
Removing intermediate container d4127754615f
---> fd635a371ac3
Step 30/32 : RUN touch $TMP_FILE && rm $TMP_FILE
---> Running in 3441491edaf9
Removing intermediate container 3441491edaf9
---> dbdf302a46e6
Step 31/32 : RUN gonymizer dump --dump-file="$TMP_FILE" -S -H "$PG_SRC_HOST" -p "$PG_SRC_PSWD" -U "$PG_SRC_USER" -P "$PG_SRC_PORT" -d "$PG_SRC_DBNAME" --schema "$PG_SRC_SCHEMA" -L "$DBG_LVL"
---> Running in 2a7de0e1e1f8
time="2020-09-08 09:55:36.938" level=debug msg="🐍 \x1b[1;32m configuration \x1b[0m 👇"
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
time="2020-09-08 09:55:36.938" level=debug msg="🐍 \x1b[1;32m configuration \x1b[0m ☝️"
time="2020-09-08 09:55:36.938" level=debug msg="os.Args: [gonymizer dump --dump-file=grossolini_as.tmp.sql -S **REMOVED** -L DEBUG]"
PFlags:
map[string]viper.FlagValue{"config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d06e0)}, "database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0c80)}, "disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0960)}, "dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1180)}, "dump.database":viper.pflagValue{flag:(*pflag.Flag)(0xc000095c20)}, "dump.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc000095900)}, "dump.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc000095cc0)}, "dump.exclude-schemas":viper.pflagValue{flag:(*pflag.Flag)(0xc000095ae0)}, "dump.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0000959a0)}, "dump.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc000095a40)}, "dump.host":viper.pflagValue{flag:(*pflag.Flag)(0xc000095b80)}, "dump.password":viper.pflagValue{flag:(*pflag.Flag)(0xc000095ea0)}, "dump.port":viper.pflagValue{flag:(*pflag.Flag)(0xc000095f40)}, "dump.row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d05a0)}, "dump.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc000095d60)}, "dump.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d00a0)}, "exclude-schemas":viper.pflagValue{flag:(*pflag.Flag)(0xc000095ae0)}, "exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0aa0)}, "exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0b40)}, "generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1040)}, "help":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1400)}, "host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0be0)}, "load.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0280)}, "load.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0140)}, "load.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d01e0)}, "load.load-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0320)}, "load.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0460)}, "load.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0500)}, "load.skip-procedures":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d03c0)}, "load.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0640)}, "log-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0780)}, "log-format":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d08c0)}, "log-level":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0820)}, "map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d10e0)}, "map.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0c80)}, "map.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0960)}, "map.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0aa0)}, "map.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0b40)}, "map.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0be0)}, "map.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0a00)}, "map.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0e60)}, "map.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0f00)}, "map.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0d20)}, "map.schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0dc0)}, "map.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0fa0)}, "password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0e60)}, "port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0f00)}, "post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d12c0)}, "process.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1180)}, "process.generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1040)}, "process.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d10e0)}, "process.post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d12c0)}, "process.processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1220)}, "process.s3-file-path":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1360)}, "processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1220)}, "row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0000)}, "s3-file-path":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d1360)}, "schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0d20)}, "schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0dc0)}, "username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001d0fa0)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
time="2020-09-08 09:55:36.938" level=debug msg="Starting gonymizer (v1.1.1, build 5, build date:2019-05-06 23:50:02 +0000 UTC)"
time="2020-09-08 09:55:36.938" level=debug msg="Go (runtime:go1.12.6) (GOMAXPROCS:2) (NumCPUs:2)\n"
time="2020-09-08 09:55:36.938" level=info msg="\x1b[1;33mEnabling log level: DEBUG\x1b[0m"
time="2020-09-08 09:55:41.963" level=info msg="🚜 \x1b[1;32mCreating dump file\x1b[0m 🚜"
time="2020-09-08 09:55:41.963" level=info msg="Dumping the following schemas: [as]"
time="2020-09-08 09:55:41.963" level=debug msg="Running command: pg_dump **REMOVED** --oids --no-owner --schema=as.*"
time="2020-09-08 09:56:34.741" level=info msg="🦄 \x1b[1;32m-- SUCCESS --\x1b[0m 🌈"
Removing intermediate container 2a7de0e1e1f8
---> 08b12d84887b
Step 32/32 : RUN gonymizer process --map-file="$SKEL_FILE" --dump-file="$TMP_FILE" --processed-file="$DUMP_FILE" --generate-seed -L "$DBG_LVL"
---> Running in a99a167a770a
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
time="2020-09-08 09:56:40.034" level=debug msg="🐍 \x1b[1;32m configuration \x1b[0m 👇"
PFlags:
map[string]viper.FlagValue{"config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac640)}, "database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acbe0)}, "disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac8c0)}, "dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad0e0)}, "dump.database":viper.pflagValue{flag:(*pflag.Flag)(0xc000079b80)}, "dump.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc000079860)}, "dump.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc000079c20)}, "dump.exclude-schemas":viper.pflagValue{flag:(*pflag.Flag)(0xc000079a40)}, "dump.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc000079900)}, "dump.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0000799a0)}, "dump.host":viper.pflagValue{flag:(*pflag.Flag)(0xc000079ae0)}, "dump.password":viper.pflagValue{flag:(*pflag.Flag)(0xc000079e00)}, "dump.port":viper.pflagValue{flag:(*pflag.Flag)(0xc000079ea0)}, "dump.row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac500)}, "dump.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc000079cc0)}, "dump.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac000)}, "exclude-schemas":viper.pflagValue{flag:(*pflag.Flag)(0xc000079a40)}, "exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001aca00)}, "exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acaa0)}, "generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acfa0)}, "help":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad360)}, "host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acb40)}, "load.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac1e0)}, "load.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac0a0)}, "load.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac140)}, "load.load-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac280)}, "load.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac3c0)}, "load.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac460)}, "load.skip-procedures":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac320)}, "load.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac5a0)}, "log-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac6e0)}, "log-format":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac820)}, "log-level":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac780)}, "map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad040)}, "map.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acbe0)}, "map.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac8c0)}, "map.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001aca00)}, "map.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acaa0)}, "map.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acb40)}, "map.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ac960)}, "map.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acdc0)}, "map.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ace60)}, "map.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acc80)}, "map.schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acd20)}, "map.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acf00)}, "password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acdc0)}, "port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ace60)}, "post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad220)}, "process.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad0e0)}, "process.generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acfa0)}, "process.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad040)}, "process.post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad220)}, "process.processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad180)}, "process.s3-file-path":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad2c0)}, "processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad180)}, "row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc000079f40)}, "s3-file-path":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ad2c0)}, "schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acc80)}, "schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acd20)}, "username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001acf00)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
time="2020-09-08 09:56:40.034" level=debug msg="🐍 \x1b[1;32m configuration \x1b[0m ☝️"
time="2020-09-08 09:56:40.034" level=debug msg="os.Args: [gonymizer process --map-file=grossolini_as.skeleton.json --dump-file=grossolini_as.tmp.sql --processed-file=grossolini_as.dump.sql --generate-seed -L DEBUG]"
time="2020-09-08 09:56:40.034" level=debug msg="Starting gonymizer (v1.1.1, build 5, build date:2019-05-06 23:50:02 +0000 UTC)"
time="2020-09-08 09:56:40.034" level=debug msg="Go (runtime:go1.12.6) (GOMAXPROCS:2) (NumCPUs:2)\n"
time="2020-09-08 09:56:40.034" level=info msg="\x1b[1;33mEnabling log level: DEBUG\x1b[0m"
time="2020-09-08 09:56:40.034" level=debug msg="s3-file-path: "
time="2020-09-08 09:56:40.034" level=debug msg="S3 URL: <nil>\tScheme: \tBucket: \tRegion: \tFile Path: "
time="2020-09-08 09:56:40.034" level=info msg="🚜 \x1b[1;32mProcessing dump file\x1b[0m 🚜"
time="2020-09-08 09:56:40.034" level=info msg="Loading map file from: grossolini_as.skeleton.json"
time="2020-09-08 09:56:40.035" level=info msg="Processing dump file: grossolini_as.tmp.sql"
time="2020-09-08 09:56:40.035" level=debug msg="Using internal number generator for seed value: 2135716918243314402"
[ ...a bunch of Schema.Table logs... ]
time="2020-09-08 09:56:54.899" level=info msg="Processing line number: 2900000"
time="2020-09-08 09:56:55.249" level=info msg="Processing line number: 3000000"
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/smithoss/gonymizer.(*LineState).parseCopyLine(0xc000373928, 0xc000260aa0, 0x13)
/usr/local/go/src/github.com/smithoss/gonymizer/generator.go:404 +0x5ac
github.com/smithoss/gonymizer.processLine(0xc00005d480, 0xc000373928, 0xc000260aa0, 0x13, 0xc000260aa0, 0x13, 0x0, 0x0, 0x0)
/usr/local/go/src/github.com/smithoss/gonymizer/generator.go:301 +0x276
github.com/smithoss/gonymizer.ProcessDumpFile(0xc00005d480, 0x7ffd256fdd75, 0x15, 0x7ffd256fdd9c, 0x16, 0x0, 0x0, 0xaca301, 0x0, 0x0)
/usr/local/go/src/github.com/smithoss/gonymizer/generator.go:200 +0x8de
main.process(0x7ffd256fdd75, 0x15, 0x7ffd256fdd4d, 0x1b, 0x7ffd256fdd9c, 0x16, 0x1, 0xc00007ea00, 0xc000157b30, 0x0)
/usr/local/go/src/github.com/smithoss/gonymizer/command/process.go:136 +0x20d
main.cliCommandProcess(0x126ea60, 0xc000060ba0, 0x0, 0x6)
/usr/local/go/src/github.com/smithoss/gonymizer/command/process.go:111 +0x830
github.com/spf13/cobra.(*Command).execute(0x126ea60, 0xc000060a80, 0x6, 0x6, 0x126ea60, 0xc000060a80)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0x126e5a0, 0x0, 0x0, 0xc00015a900)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
/usr/local/go/src/github.com/smithoss/gonymizer/command/main.go:116 +0x32
The command '/bin/sh -c gonymizer process --map-file="$SKEL_FILE" --dump-file="$TMP_FILE" --processed-file="$DUMP_FILE" --generate-seed -L "$DBG_LVL"' returned a non-zero code: 2
Here is the output using the junkert/gonymizer image:
Step 25/32 : RUN pg_dump --version && pg_restore --version && gonymizer version
---> Running in dee19173bf66
pg_dump (PostgreSQL) 12.3
pg_restore (PostgreSQL) 12.3
gonymizer (v1.2.0, build 10, build date:2019-07-31 17:23:35 +0000 UTC)
Go (runtime:go1.13.15) (GOMAXPROCS:2) (NumCPUs:2)
Removing intermediate container dee19173bf66
---> a344329f4c4b
Step 26/32 : RUN apk update
---> Running in 5a2dcd65cd28
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.0-295-gaaa7f43dbe [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-301-g6a4ffc91b5 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12743 distinct packages available
Removing intermediate container 5a2dcd65cd28
---> c50cfe680df3
Step 27/32 : RUN apk add jq
---> Running in 7ee73a81b6a9
(1/2) Installing oniguruma (6.9.5-r1)
(2/2) Installing jq (1.6-r1)
Executing busybox-1.31.1-r19.trigger
OK: 143 MiB in 34 packages
Removing intermediate container 7ee73a81b6a9
---> 4a6b2d3c0e5c
Step 28/32 : RUN touch $SKEL_FILE && rm $SKEL_FILE
---> Running in f7c9b3e50d61
Removing intermediate container f7c9b3e50d61
---> 8a9bb573a6be
Step 29/32 : RUN echo '{"DBName":"grossolini_as","SchemaPrefix":"","Seed":0,"ColumnMaps":[{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_login","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":16,"IsNullable":true,"Processors":[{"Name":"AlphaNumericScrambler","Max":0,"Min":0,"Variance":0,"Comment":""}]},{"Comment":"","TableSchema":"as","TableName":"as_streams_search","ColumnName":"v3_deprecated_password","DataType":"character varying","ParentSchema":"","ParentTable":"","ParentColumn":"","OrdinalPosition":17,"IsNullable":true,"Processors":[{"Name":"ScrubString","Max":0,"Min":0,"Variance":0,"Comment":""}]}]}' | jq > $SKEL_FILE
---> Running in e9f771655822
Removing intermediate container e9f771655822
---> 976c25ed8ce8
Step 30/32 : RUN touch $TMP_FILE && rm $TMP_FILE
---> Running in f61f78f618a5
Removing intermediate container f61f78f618a5
---> 1b0f4064cfd2
Step 31/32 : RUN gonymizer dump --dump-file="$TMP_FILE" -S -H "$PG_SRC_HOST" -p "$PG_SRC_PSWD" -U "$PG_SRC_USER" -P "$PG_SRC_PORT" -d "$PG_SRC_DBNAME" --schema "$PG_SRC_SCHEMA" -L "$DBG_LVL"
---> Running in d84f19600f4a
time="2020-09-08 10:03:48.473" level=debug msg="os.Args: [gonymizer dump --dump-file=grossolini_as.tmp.sql -S **REMOVED** -L DEBUG]"
time="2020-09-08 10:03:48.473" level=debug msg="🐍 configuration 👇"
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{"config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3220)}, "database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a37c0)}, "disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a34a0)}, "dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3cc0)}, "dump.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2780)}, "dump.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2460)}, "dump.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2820)}, "dump.exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2640)}, "dump.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2500)}, "dump.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a25a0)}, "dump.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a26e0)}, "dump.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2a00)}, "dump.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2aa0)}, "dump.row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a30e0)}, "dump.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a28c0)}, "dump.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2be0)}, "exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2640)}, "exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a35e0)}, "exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3680)}, "generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3b80)}, "help":viper.pflagValue{flag:(*pflag.Flag)(0xc0001f2140)}, "host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3720)}, "inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3d60)}, "load.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2dc0)}, "load.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2c80)}, "load.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2d20)}, "load.load-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2e60)}, "load.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2fa0)}, "load.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3040)}, "load.skip-procedures":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2f00)}, "load.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3180)}, "local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001f20a0)}, "log-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a32c0)}, "log-format":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3400)}, "log-level":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3360)}, "map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3c20)}, "map.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a37c0)}, "map.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a34a0)}, "map.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a35e0)}, "map.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3680)}, "map.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3720)}, "map.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3540)}, "map.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a39a0)}, "map.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3a40)}, "map.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3860)}, "map.schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3900)}, "map.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3ae0)}, "password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a39a0)}, "port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3a40)}, "post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3f40)}, "pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3ea0)}, "process.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3cc0)}, "process.generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3b80)}, "process.inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3d60)}, "process.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3c20)}, "process.post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3f40)}, "process.pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3ea0)}, "process.processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3e00)}, "processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3e00)}, "row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a2b40)}, "s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001f2000)}, "schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3860)}, "schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3900)}, "upload.local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001f20a0)}, "upload.s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001f2000)}, "username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001a3ae0)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
time="2020-09-08 10:03:48.473" level=debug msg="🐍 configuration ☝️"
time="2020-09-08 10:03:48.473" level=debug msg="Starting gonymizer (v1.2.0, build 10, build date: 2019-07-31 17:23:35 +0000 UTC)"
time="2020-09-08 10:03:48.473" level=debug msg="Go (runtime: go1.13.15) (GOMAXPROCS: 2) (NumCPUs: 2)
"
time="2020-09-08 10:03:48.473" level=info msg="Enabling log level: DEBUG"
time="2020-09-08 10:03:48.506" level=info msg="🚜 Creating dump file 🚜"
time="2020-09-08 10:03:48.506" level=debug msg="Running command: pg_dump --oids --no-owner --schema=as.* -f grossolini_as.tmp.sql **REMOVED**"
time="2020-09-08 10:03:48.509" level=error msg="exit status 1"
time="2020-09-08 10:03:48.509" level=debug msg="name: pg_dump"
time="2020-09-08 10:03:48.509" level=debug msg="arg: [--oids --no-owner --schema=as.* -f grossolini_as.tmp.sql **REMOVED**]"
time="2020-09-08 10:03:48.509" level=debug msg="errBytes:
=====================
pg_dump: unrecognized option: oids
Try "pg_dump --help" for more information.
=====================
"
time="2020-09-08 10:03:48.509" level=debug msg="errBytes:
=====================
pg_dump: unrecognized option: oids
Try "pg_dump --help" for more information.
=====================
"
time="2020-09-08 10:03:48.509" level=error msg="STDOUT: "
time="2020-09-08 10:03:48.509" level=error msg="STDERR: pg_dump: unrecognized option: oids
Try "pg_dump --help" for more information.
"
time="2020-09-08 10:03:48.509" level=error msg="exit status 1"
time="2020-09-08 10:03:48.509" level=error msg="exit status 1"
time="2020-09-08 10:03:48.509" level=error msg="❌ Gonymizer did not exit properly. See above for errors ❌"
The command '/bin/sh -c gonymizer dump --dump-file="$TMP_FILE" -S -H "$PG_SRC_HOST" -p "$PG_SRC_PSWD" -U "$PG_SRC_USER" -P "$PG_SRC_PORT" -d "$PG_SRC_DBNAME" --schema "$PG_SRC_SCHEMA" -L "$DBG_LVL"' returned a non-zero code: 1
@GuillaumeRossolini which version are you currently using? Did you build this off of the master branch or off of a versioned branch? I recommend trying version 1.2.3 off of Github and building manually. Docker builds may be broken and I'll need to look into this when I have some time.
@junkert I am using the images that are published on docker hub. The dockerfile I pasted is pretty much verbatim, just the database credentials removed. The versions should be in the post above too, or were you asking for something else?
Let me know if I can provide anything else, run tests etc.
I'm sure a newer version in the docker image would help, but I tried building from source and I didn't understand how to set it up.
By the way the tool looks promising, thanks for sharing!
Oh right sorry I missed that. Let me see if I can do some digging to see what is going on.
I pasted the output from both docker hub images, if it helps. The second one baffles me, version is more recent but seems to not support oids? Not sure what I did wrong.
@GuillaumeRossolini what version of postgresql are you hitting? pg_dump in version 12 got rid of --oids https://www.postgresql.org/docs/12/app-pgdump.html.
Since we're still running version 11 internally, we've had to manually pin the cli on our internal Dockerfile with the following commands:
# Installs and pins postgresql to v11
RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.10/main' >> /etc/apk/repositories
RUN apk update && apk add --no-cache postgresql==11.7-r0 postgresql-client==11.7-r0
I am testing against a v9.4 server at the moment, and this will be it for pg_restore for the foreseeable future.
The target server for pg_dump is v10 (not sure the lower version nb), however it's the same v9.4 server at the moment.
@wavemoran I don't have an issue with the oids in the smithoss/gonymizer image. The issue with the oids is in the junkert/gonymizer image (which has gonymizer v1.2.0 instead of v1.1.1), and in that one I couldn't use your tip:
Step 25/34 : RUN pg_dump --version && pg_restore --version && gonymizer version
---> Running in c8cec44b4113
pg_dump (PostgreSQL) 12.3
pg_restore (PostgreSQL) 12.3
gonymizer (v1.2.0, build 10, build date:2019-07-31 17:23:35 +0000 UTC)
Go (runtime:go1.13.15) (GOMAXPROCS:2) (NumCPUs:2)
Removing intermediate container c8cec44b4113
---> 2cd5355125c9
Step 26/34 : RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.10/main' >> /etc/apk/repositories
---> Running in 6df20f7e21ec
Removing intermediate container 6df20f7e21ec
---> 0f1e50494209
Step 27/34 : RUN apk update
---> Running in 7d768ca115e6
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
v3.12.0-306-g6984aa70ea [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-306-g6984aa70ea [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
v3.10.5-84-g2392c9192e [http://dl-cdn.alpinelinux.org/alpine/v3.10/main]
OK: 18412 distinct packages available
Removing intermediate container 7d768ca115e6
---> 03369bf23cb8
Step 28/34 : RUN apk add jq
---> Running in 7bed1a0b6bfc
(1/2) Installing oniguruma (6.9.5-r1)
(2/2) Installing jq (1.6-r1)
Executing busybox-1.31.1-r19.trigger
OK: 143 MiB in 34 packages
Removing intermediate container 7bed1a0b6bfc
---> 61b98c649ad4
Step 29/34 : RUN apk add --no-cache postgresql==11.3-r2 postgresql-client==11.3-r2
---> Running in 0cc66168ad8a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
postgresql-client-12.3-r2:
breaks: world[postgresql-client=11.3-r2]
satisfies: postgresql-12.3-r2[postgresql-client]
postgresql-12.3-r2:
breaks: world[postgresql=11.3-r2]
The command '/bin/sh -c apk add --no-cache postgresql==11.3-r2 postgresql-client==11.3-r2' returned a non-zero code: 2
Yeah, looks like Alpine deprecated the 11.7 package. We updated our dockerfile to this
# Installs and pins postgresql to v11
RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.10/main' >> /etc/apk/repositories
RUN apk update && apk add --no-cache postgresql==11.9-r0 postgresql-client==11.9-r0
@wavemoran I appreciate the help but the alternative image from @junkert seems to require postgresql-client-12.3-r2 so I still can't use it, I have no idea if it would solve the main issue, and anyway the main image from smithoss doesn't block at this stage but much later. As far as I understand it, the issue is a log output failing to alert me (panic: runtime error) about the core issue with my use case, probably a misconfiguration on my part.
At the moment I've sidestepped the oids issue by copying the projected pg_dump command from the DEBUG output, removed --oids from the command line, and used that instead of gonymizer dump. I'll update here if gonymizer v1.2.0 helps with the initial issue, reported for v1.1.1.
@junkert FYI gonymizer v1.2.0 has a similar issue as v1.1.1:
Step 25/34 : RUN pg_dump --version && pg_restore --version && gonymizer version
---> Running in 60e557612026
pg_dump (PostgreSQL) 12.3
pg_restore (PostgreSQL) 12.3
gonymizer (v1.2.0, build 10, build date:2019-07-31 17:23:35 +0000 UTC)
Go (runtime:go1.13.15) (GOMAXPROCS:2) (NumCPUs:2)
[...]
Step 33/34 : RUN gonymizer process --map-file="$SKEL_FILE" --dump-file="$TMP_FILE" --processed-file="$DUMP_FILE" --generate-seed -L "$DBG_LVL"
---> Running in 4926174a63fc
time="2020-09-10 08:50:36.381" level=info msg="Enabling log level: INFO"
time="2020-09-10 08:50:36.381" level=info msg="🚜 Processing dump file 🚜"
time="2020-09-10 08:50:36.382" level=info msg="Loading map file from: grossolini_as.skeleton.json"
time="2020-09-10 08:50:36.397" level=info msg="Processing dump file: grossolini_as.tmp.sql"
time="2020-09-10 08:50:37.257" level=info msg="Processing line number: 100000"
time="2020-09-10 08:50:39.939" level=info msg="Processing line number: 200000"
[...]
time="2020-09-10 09:02:12.563" level=info msg="Processing line number: 2900000"
time="2020-09-10 09:02:40.195" level=info msg="Processing line number: 3000000"
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/smithoss/gonymizer.(*LineState).parseCopyLine(0xc0003fd988, 0xc0004d7180, 0x13)
/tmp/gonymizer/generator.go:425 +0x5ba
github.com/smithoss/gonymizer.processLine(0xc00007b300, 0xc0003fd988, 0xc0004d7180, 0x13, 0xc0004d7180, 0x13, 0x0, 0x0, 0x0)
/tmp/gonymizer/generator.go:317 +0x272
github.com/smithoss/gonymizer.ProcessDumpFile(0xc00007b300, 0x7ffce8c87d60, 0x16, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffce8c87d39, ...)
/tmp/gonymizer/generator.go:212 +0x829
main.process(0x7ffce8c87d39, 0x15, 0x7ffce8c87d11, 0x1b, 0x7ffce8c87d60, 0x16, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/gonymizer/cmd/process.go:137 +0x37b
main.cliCommandProcess(0x12a6300, 0xc00005cc60, 0x0, 0x6)
/tmp/gonymizer/cmd/process.go:97 +0x552
github.com/spf13/cobra.(*Command).execute(0x12a6300, 0xc00005cb40, 0x6, 0x6, 0x12a6300, 0xc00005cb40)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x12a5e00, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/tmp/gonymizer/cmd/main.go:121 +0x31
The command '/bin/sh -c gonymizer process --map-file="$SKEL_FILE" --dump-file="$TMP_FILE" --processed-file="$DUMP_FILE" --generate-seed -L "$DBG_LVL"' returned a non-zero code: 2
@junkert we identified this particular issue in our data. It could be exported and reimported safely with pg_dump/pg_restore, but gonymizer was unable to process these rows.
This issue lies in overly escaped fields, like \\\\\\"stuff\\\\\\". We should fix that on our side.
However, when skipping this data in the dump, gonymizer fails at another point with the same error message as before. This time the erroneous data is harder to pinpoint, my guess is either “...” (the oblique versions to the double quotes character) or the &\n sequence. Both of there are found in our data in the vicinity of the last debug line from gonymizer (assuming gonymizer uses the output from pg_dump, with comparable line numbers), but I may be mistaken.
@GuillaumeRossolini what happens if you disable debug mode? Maybe try a higher level of debug such as info
This looks like it may be a simple issue of the console not being able to display the debug output here:
https://github.com/smithoss/gonymizer/blob/v1.2.0/generator.go#L418-L427
Let me know if this gets you any further.
@junkert The last log I pasted here, from Sept 10, was with not debug but rather info. The bug doesn't look like it's in the console debugs, but rather the parsing of the line. Since the data apparently contains outlandish escaping or characters (in this case), perhaps it breaks the parsing tool where pg_restore wouldn't be confused. I found a way around this (for my use case) because we happen not to need the two tables where this happens, but maybe you'd like to look into it anyway.
@GuillaumeRossolini these parse errors are really hard to fix without concrete examples. Can you build me a mock SQL dump that I can load into PSQL with a couple rows of data that I can use to test against?
@junkert I found 1 tuple that consistently breaks gonimyzer process:
(I included the grepped skeleton file to show that no processor is configured for that table)
Step 47/52 : RUN pg_dump postgres://${PG_SRC_USER}:${PG_SRC_PSWD}@${PG_SRC_HOST}:${PG_SRC_PORT}/${PG_SRC_DBNAME}?sslmode=disable --no-owner -no-sync --schema="${PG_SRC_SCHEMA}.*" --table="as.grtmp_locales" >$TMP_DUMP_FILE
---> Running in a0a08ced0358
---> aa5a79f745d0
Removing intermediate container a0a08ced0358
Step 48/52 : RUN cat "$IVS_SKEL_FILE" | grep "grtmp_locales" | wc -l
---> Running in 2fbdde8b55df
0
---> 45076be684c1
Removing intermediate container 2fbdde8b55df
Step 49/52 : RUN cat "$TMP_DUMP_FILE"
---> Running in bb84daa14f12
--
-- PostgreSQL database dump
--
-- Dumped from database version 9.4.18
-- Dumped by pg_dump version 12.3
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
--
-- Name: grtmp_locales; Type: TABLE; Schema: as; Owner: -
--
CREATE TABLE "as".grtmp_locales (
item_key character varying(128),
item_comment text
);
--
-- Data for Name: grtmp_locales; Type: TABLE DATA; Schema: as; Owner: -
--
COPY "as".grtmp_locales (item_key, item_comment) FROM stdin;
COPYRIGHTIVS
\.
--
-- PostgreSQL database dump complete
--
---> 42e756e7320b
Removing intermediate container bb84daa14f12
Step 50/52 : RUN gonymizer process --generate-seed --map-file="$IVS_SKEL_FILE" --dump-file="$TMP_DUMP_FILE" --processed-file="$ANON_DUMP_FILE" -L "$DBG_LVL"
---> Running in bddb9ae0ffe9
time="2020-09-29 09:42:27.876" level=debug msg="os.Args: [gonymizer process --generate-seed --map-file=/tmp/grossolini_as.skel.json --dump-file=/tmp/grossolini_as.orig.sql --processed-file=/tmp/grossolini_as.anon.sql -L DEBUG]"
time="2020-09-29 09:42:27.876" level=debug msg="🐍 configuration 👇"
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{"config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9220)}, "database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b97c0)}, "disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b94a0)}, "dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9cc0)}, "dump.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8780)}, "dump.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8460)}, "dump.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8820)}, "dump.exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8640)}, "dump.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8500)}, "dump.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b85a0)}, "dump.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b86e0)}, "dump.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8a00)}, "dump.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8aa0)}, "dump.row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b90e0)}, "dump.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b88c0)}, "dump.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8be0)}, "exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8640)}, "exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b95e0)}, "exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9680)}, "generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9b80)}, "help":viper.pflagValue{flag:(*pflag.Flag)(0xc000206140)}, "host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9720)}, "inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9d60)}, "load.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8dc0)}, "load.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8c80)}, "load.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8d20)}, "load.load-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8e60)}, "load.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8fa0)}, "load.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9040)}, "load.skip-procedures":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8f00)}, "load.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9180)}, "local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0002060a0)}, "log-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b92c0)}, "log-format":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9400)}, "log-level":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9360)}, "map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9c20)}, "map.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b97c0)}, "map.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b94a0)}, "map.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b95e0)}, "map.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9680)}, "map.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9720)}, "map.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9540)}, "map.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b99a0)}, "map.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9a40)}, "map.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9860)}, "map.schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9900)}, "map.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9ae0)}, "password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b99a0)}, "port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9a40)}, "post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9f40)}, "pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9ea0)}, "process.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9cc0)}, "process.generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9b80)}, "process.inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9d60)}, "process.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9c20)}, "process.post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9f40)}, "process.pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9ea0)}, "process.processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9e00)}, "processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9e00)}, "row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b8b40)}, "s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc000206000)}, "schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9860)}, "schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9900)}, "upload.local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0002060a0)}, "upload.s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc000206000)}, "username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001b9ae0)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
time="2020-09-29 09:42:27.877" level=debug msg="🐍 configuration ☝️"
time="2020-09-29 09:42:27.877" level=debug msg="Starting gonymizer (v1.2.0, build 10, build date: 2019-07-31 17:23:35 +0000 UTC)"
time="2020-09-29 09:42:27.877" level=debug msg="Go (runtime: go1.13.15) (GOMAXPROCS: 4) (NumCPUs: 4)
"
time="2020-09-29 09:42:27.877" level=info msg="Enabling log level: DEBUG"
time="2020-09-29 09:42:27.877" level=info msg="🚜 Processing dump file 🚜"
time="2020-09-29 09:42:27.877" level=info msg="Loading map file from: /tmp/grossolini_as.skel.json"
time="2020-09-29 09:42:27.878" level=info msg="Processing dump file: /tmp/grossolini_as.orig.sql"
time="2020-09-29 09:42:27.878" level=debug msg="Using internal number generator for seed value: 506506504246908844"
time="2020-09-29 09:42:27.893" level=debug msg="
====================================================================================================================
Schema.Table: "as".grtmp_locales
Line number: 35
Is a row: true
Columns: item_key, item_comment
===================================================================================================================="
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/smithoss/gonymizer.(*LineState).parseCopyLine(0xc0001e5988, 0xc00023cbd0, 0xe)
/tmp/gonymizer/generator.go:425 +0x5ba
github.com/smithoss/gonymizer.processLine(0xc00007d300, 0xc0001e5988, 0xc00023cbd0, 0xe, 0xc00023cbd0, 0xe, 0x0, 0x0, 0x0)
/tmp/gonymizer/generator.go:317 +0x272
github.com/smithoss/gonymizer.ProcessDumpFile(0xc00007d300, 0x7ffdd6bb4c70, 0x1b, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffdd6bb4c43, ...)
/tmp/gonymizer/generator.go:212 +0x829
main.process(0x7ffdd6bb4c43, 0x1b, 0x7ffdd6bb4c1a, 0x1c, 0x7ffdd6bb4c70, 0x1b, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/gonymizer/cmd/process.go:137 +0x37b
main.cliCommandProcess(0x12a6300, 0xc00005ec60, 0x0, 0x6)
/tmp/gonymizer/cmd/process.go:97 +0x552
github.com/spf13/cobra.(*Command).execute(0x12a6300, 0xc00005eb40, 0x6, 0x6, 0x12a6300, 0xc00005eb40)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x12a5e00, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/tmp/gonymizer/cmd/main.go:121 +0x31
The command '/bin/sh -c gonymizer process --generate-seed --map-file="$IVS_SKEL_FILE" --dump-file="$TMP_DUMP_FILE" --processed-file="$ANON_DUMP_FILE" -L "$DBG_LVL"' returned a non-zero code: 2
I couldn't see anythng wrong with the data. The second field is nullable, and that particular tuple has an empty string in this field. To make sure no invisible characters are present, I tried again after manually updating the field to the empty string, but the same error occurs.
As it happens, we had this error on 2 tables of the database, and the second table also has the same value "COPYRIGHTIVS" in one of the fields. This seems to be what trips up gonimyzer process.
Step 48/52 : RUN cat "$IVS_SKEL_FILE" | grep "grtmp_locales" | wc -l
---> Running in 439aa2a41523
0
---> 8d5d2a0d7c2b
Removing intermediate container 439aa2a41523
Step 49/52 : RUN cat "$TMP_DUMP_FILE"
---> Running in bd12ee9d5392
--
-- PostgreSQL database dump
--
-- Dumped from database version 9.4.18
-- Dumped by pg_dump version 12.3
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
--
-- Name: grtmp_locales2; Type: TABLE; Schema: as; Owner: -
--
CREATE TABLE "as".grtmp_locales2 (
item_key character varying(128),
lang_key character varying(10),
value text
);
--
-- Data for Name: grtmp_locales2; Type: TABLE DATA; Schema: as; Owner: -
--
COPY "as".grtmp_locales2 (item_key, lang_key, value) FROM stdin;
COPYRIGHTIVS es-ar
COPYRIGHTIVS zh-hk
COPYRIGHTIVS th-th
COPYRIGHTIVS en-au
COPYRIGHTIVS zh-tw
COPYRIGHTIVS es-mx
COPYRIGHTIVS en-ie
COPYRIGHTIVS de-ch
COPYRIGHTIVS de-be
COPYRIGHTIVS de-at
COPYRIGHTIVS fr-be
COPYRIGHTIVS fr-ch
COPYRIGHTIVS it-ch
COPYRIGHTIVS nl-be
COPYRIGHTIVS en-nz
COPYRIGHTIVS en-cn
COPYRIGHTIVS en-my
COPYRIGHTIVS en-sg
COPYRIGHTIVS en-th
COPYRIGHTIVS bg-bg
COPYRIGHTIVS ca-es
COPYRIGHTIVS cs-cz
COPYRIGHTIVS da-dk
COPYRIGHTIVS de-de
COPYRIGHTIVS en-id
COPYRIGHTIVS en-gb
COPYRIGHTIVS en-ca
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fi-fi
COPYRIGHTIVS fr-ca
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS hr-hr
COPYRIGHTIVS hu-hu
COPYRIGHTIVS id-id
COPYRIGHTIVS it-it
COPYRIGHTIVS ja-jp
COPYRIGHTIVS nl-nl
COPYRIGHTIVS no-no
COPYRIGHTIVS pl-pl
COPYRIGHTIVS pt-br
COPYRIGHTIVS pt-pt
COPYRIGHTIVS ro-ro
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS sk-sk
COPYRIGHTIVS sl-si
COPYRIGHTIVS es-es
COPYRIGHTIVS sr-rs
COPYRIGHTIVS sv-se
COPYRIGHTIVS tr-tr © IVS 1996-2005
COPYRIGHTIVS zh-cn
COPYRIGHTIVS el-gr
COPYRIGHTIVS es-ar
COPYRIGHTIVS zh-hk
COPYRIGHTIVS th-th
COPYRIGHTIVS en-au
COPYRIGHTIVS zh-tw
COPYRIGHTIVS es-mx
COPYRIGHTIVS en-ie
COPYRIGHTIVS de-ch
COPYRIGHTIVS de-be
COPYRIGHTIVS de-at
COPYRIGHTIVS fr-be
COPYRIGHTIVS fr-ch
COPYRIGHTIVS it-ch
COPYRIGHTIVS nl-be
COPYRIGHTIVS en-nz
COPYRIGHTIVS en-cn
COPYRIGHTIVS en-my
COPYRIGHTIVS en-sg
COPYRIGHTIVS en-th
COPYRIGHTIVS bg-bg
COPYRIGHTIVS ca-es
COPYRIGHTIVS cs-cz
COPYRIGHTIVS da-dk
COPYRIGHTIVS de-de
COPYRIGHTIVS en-id
COPYRIGHTIVS en-gb
COPYRIGHTIVS en-ca
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fi-fi
COPYRIGHTIVS fr-ca
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS hr-hr
COPYRIGHTIVS hu-hu
COPYRIGHTIVS id-id
COPYRIGHTIVS it-it
COPYRIGHTIVS ja-jp
COPYRIGHTIVS nl-nl
COPYRIGHTIVS no-no
COPYRIGHTIVS pl-pl
COPYRIGHTIVS pt-br
COPYRIGHTIVS pt-pt
COPYRIGHTIVS ro-ro
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS sk-sk
COPYRIGHTIVS sl-si
COPYRIGHTIVS es-es
COPYRIGHTIVS sr-rs
COPYRIGHTIVS sv-se
COPYRIGHTIVS tr-tr © IVS 1996-2005
COPYRIGHTIVS zh-cn
COPYRIGHTIVS zh-hk
COPYRIGHTIVS th-th
COPYRIGHTIVS en-au
COPYRIGHTIVS zh-tw
COPYRIGHTIVS es-mx
COPYRIGHTIVS en-ie
COPYRIGHTIVS de-ch
COPYRIGHTIVS de-be
COPYRIGHTIVS de-at
COPYRIGHTIVS fr-be
COPYRIGHTIVS fr-ch
COPYRIGHTIVS it-ch
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS tr-tr © IVS 1996-2005
COPYRIGHTIVS nl-be
COPYRIGHTIVS en-nz
COPYRIGHTIVS en-cn
COPYRIGHTIVS en-my
COPYRIGHTIVS en-sg
COPYRIGHTIVS en-th
COPYRIGHTIVS bg-bg
COPYRIGHTIVS ca-es
COPYRIGHTIVS cs-cz
COPYRIGHTIVS da-dk
COPYRIGHTIVS el-gr
COPYRIGHTIVS de-de
COPYRIGHTIVS en-id
COPYRIGHTIVS en-gb
COPYRIGHTIVS en-ca
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fi-fi
COPYRIGHTIVS fr-ca
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS hr-hr
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS tr-tr © IVS 1996-2005
COPYRIGHTIVS hu-hu
COPYRIGHTIVS id-id
COPYRIGHTIVS it-it
COPYRIGHTIVS ja-jp
COPYRIGHTIVS nl-nl
COPYRIGHTIVS no-no
COPYRIGHTIVS pl-pl
COPYRIGHTIVS pt-br
COPYRIGHTIVS pt-pt
COPYRIGHTIVS ro-ro
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS sk-sk
COPYRIGHTIVS sl-si
COPYRIGHTIVS es-es
COPYRIGHTIVS sr-rs
COPYRIGHTIVS sv-se
COPYRIGHTIVS tr-tr © IVS 1996-2005
COPYRIGHTIVS zh-cn
COPYRIGHTIVS el-gr
COPYRIGHTIVS es-ar
COPYRIGHTIVS en-us © IVS 1996-2005
COPYRIGHTIVS fr-fr © IVS 1996-2005
COPYRIGHTIVS ru-ru © IVS 1996-2005
COPYRIGHTIVS tr-tr © IVS 1996-2005
\.
--
-- PostgreSQL database dump complete
--
---> 53674424f498
Removing intermediate container bd12ee9d5392
Step 50/52 : RUN gonymizer process --generate-seed --map-file="$IVS_SKEL_FILE" --dump-file="$TMP_DUMP_FILE" --processed-file="$ANON_DUMP_FILE" -L "$DBG_LVL"
---> Running in 59cd959a4661
time="2020-09-29 09:58:37.489" level=debug msg="os.Args: [gonymizer process --generate-seed --map-file=/tmp/grossolini_as.skel.json --dump-file=/tmp/grossolini_as.orig.sql --processed-file=/tmp/grossolini_as.anon.sql -L DEBUG]"
time="2020-09-29 09:58:37.489" level=debug msg="🐍 configuration 👇"
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{"config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf2c0)}, "database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf860)}, "disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf540)}, "dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfd60)}, "dump.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce820)}, "dump.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce500)}, "dump.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce8c0)}, "dump.exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce6e0)}, "dump.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce5a0)}, "dump.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce640)}, "dump.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce780)}, "dump.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ceaa0)}, "dump.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ceb40)}, "dump.row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf180)}, "dump.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce960)}, "dump.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cec80)}, "exclude-schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ce6e0)}, "exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf680)}, "exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf720)}, "generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfc20)}, "help":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e1e0)}, "host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf7c0)}, "inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfe00)}, "load.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cee60)}, "load.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001ced20)}, "load.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cedc0)}, "load.load-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cef00)}, "load.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf040)}, "load.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf0e0)}, "load.skip-procedures":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cefa0)}, "load.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf220)}, "local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e140)}, "log-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf360)}, "log-format":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf4a0)}, "log-level":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf400)}, "map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfcc0)}, "map.database":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf860)}, "map.disable-ssl":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf540)}, "map.exclude-table":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf680)}, "map.exclude-table-data":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf720)}, "map.host":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf7c0)}, "map.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf5e0)}, "map.password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfa40)}, "map.port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfae0)}, "map.schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf900)}, "map.schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf9a0)}, "map.username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfb80)}, "password":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfa40)}, "port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfae0)}, "post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e000)}, "pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cff40)}, "process.dump-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfd60)}, "process.generate-seed":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfc20)}, "process.inclusive":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfe00)}, "process.map-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfcc0)}, "process.post-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e000)}, "process.pre-process-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cff40)}, "process.processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfea0)}, "processed-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfea0)}, "row-count-file":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cebe0)}, "s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e0a0)}, "schema":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf900)}, "schema-prefix":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cf9a0)}, "upload.local-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e140)}, "upload.s3-file":viper.pflagValue{flag:(*pflag.Flag)(0xc00021e0a0)}, "username":viper.pflagValue{flag:(*pflag.Flag)(0xc0001cfb80)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
time="2020-09-29 09:58:37.489" level=debug msg="🐍 configuration ☝️"
time="2020-09-29 09:58:37.489" level=debug msg="Starting gonymizer (v1.2.0, build 10, build date: 2019-07-31 17:23:35 +0000 UTC)"
time="2020-09-29 09:58:37.489" level=debug msg="Go (runtime: go1.13.15) (GOMAXPROCS: 4) (NumCPUs: 4)
"
time="2020-09-29 09:58:37.489" level=info msg="Enabling log level: DEBUG"
time="2020-09-29 09:58:37.489" level=info msg="🚜 Processing dump file 🚜"
time="2020-09-29 09:58:37.490" level=info msg="Loading map file from: /tmp/grossolini_as.skel.json"
time="2020-09-29 09:58:37.491" level=info msg="Processing dump file: /tmp/grossolini_as.orig.sql"
time="2020-09-29 09:58:37.492" level=debug msg="Using internal number generator for seed value: -434307737843501813"
time="2020-09-29 09:58:37.492" level=debug msg="
====================================================================================================================
Schema.Table: "as".grtmp_locales2
Line number: 36
Is a row: true
Columns: item_key, lang_key, value
===================================================================================================================="
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
github.com/smithoss/gonymizer.(*LineState).parseCopyLine(0xc0001fb988, 0xc0000a33c0, 0x14)
/tmp/gonymizer/generator.go:425 +0x5ba
github.com/smithoss/gonymizer.processLine(0xc0000a92c0, 0xc0001fb988, 0xc0000a33c0, 0x14, 0xc0000a33c0, 0x14, 0x0, 0x0, 0x0)
/tmp/gonymizer/generator.go:317 +0x272
github.com/smithoss/gonymizer.ProcessDumpFile(0xc0000a92c0, 0x7ffe35625c70, 0x1b, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffe35625c43, ...)
/tmp/gonymizer/generator.go:212 +0x829
main.process(0x7ffe35625c43, 0x1b, 0x7ffe35625c1a, 0x1c, 0x7ffe35625c70, 0x1b, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/gonymizer/cmd/process.go:137 +0x37b
main.cliCommandProcess(0x12a6300, 0xc0000a6c00, 0x0, 0x6)
/tmp/gonymizer/cmd/process.go:97 +0x552
github.com/spf13/cobra.(*Command).execute(0x12a6300, 0xc0000a6ae0, 0x6, 0x6, 0x12a6300, 0xc0000a6ae0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0x12a5e00, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
main.main()
/tmp/gonymizer/cmd/main.go:121 +0x31
The command '/bin/sh -c gonymizer process --generate-seed --map-file="$IVS_SKEL_FILE" --dump-file="$TMP_DUMP_FILE" --processed-file="$ANON_DUMP_FILE" -L "$DBG_LVL"' returned a non-zero code: 2
I tried with and without values in this last table, and the same always happens.
Might it be because the first field's value starts with the substring "COPY"?