Decryption fails for some data

[philicious]

I have encrypted a json file but cannot decrypt it again and get the error Could not marshal tree: No binary data found in tree

Unfortunately I cannot share the json contents. Its a Google Service Account private key. However its likely it will also fail for other json files of this type as they have the same format and similar data.

sops 3.0.5 (latest)

[autrilla]

I can't reproduce this with a Google Service Account private key I just created.

[glerchundi]

Same here

[glerchundi]

I was able to fix this error by adding explicitly the output format:

$> sops -d file.json.enc
Error dumping file: No binary data found in tree
$> sops --output-type json -d file.json.enc
{
...
}

[autrilla]

Yeah, that's not an issue with SOPS. If the extension is .enc, SOPS will assume it's a binary file and try to decrypt it as such.

@philicious was this same thing happening to you?

[glerchundi]

@autrilla it makes sense but if you think about it people expects the same to be happening in the process of encrypting and decrypting without specifying any kind of format.

I mean, i would expect this to be working with no issues at all:

$> sops --encrypt a.json > a.json.enc
$> sops --decrypt a.json.enc

Maybe include the input format inside the spec?

[autrilla]

I understand that this is your expectation, and it makes sense. However, there isn't really a good way for us to tell that the file is a json file if the extension is .enc. We could add a special case that checks if the file name is .enc and then checks for the extension before that.

The examples use *.enc.json instead of *.json.enc to avoid this issue.

Maybe include the input format inside the spec?

I'm not sure what you mean by this

[philicious]

@autrilla I dont recall exactly but it might have very well been the case that my encrypted file also had .enc extension

I would like to second @glerchundi suggestion to save the input format in the encrypted file so decryption will work seamlessly without having the user to add flags for the format.

imho, an encryption tool shouldnt require the user to care about the format. The tool can use different approaches internally but to the user it should be transparent

[autrilla]

Unfortunately what you suggest isn't really possible, because we'd need to parse the file in order to read the file format, and we can't parse the file unless we know the file format.

I think the best solution here is to just improve the error message.

[jvehent]

We could try to read the first byte of a file and if it matches { or [ then attempt to decrypt as json, then fall back to binary.

[autrilla]

@jvehent this would work right now but I think it will be very messy as we add more formats. Not sure JSON will be the only format that starts with {, unfortunately.

[jvehent]

My point is that, in the absence of --input-type, it seems reasonable to me to try a variety of formats before failing with an error. It's similar to trying multiple decryption keys.

[glerchundi]

Maybe include the input format inside the spec?

I'm not sure what you mean by this

i was talking about putting something inside the sops object like 'format' or whatever to aid decoder. Does it makes sense?

[jvehent]

@glerchundi That doesn't work as you'd need to know how to parse the file before being able to access the format value, at which point you no longer need it because you already parsed the file.

[glerchundi]

@jvehent correct me if i'm wrong but the encoded format is always the same, right?

{
	"data": <redacted>
	"sops": {
		"kms": null,
		"gcp_kms": <redacted>,
		"azure_kv": null,
		"lastmodified": "2018-09-21T07:24:06Z",
		"mac": <redacted>,
		"pgp": null,
		"unencrypted_suffix": "_unencrypted",
		"version": "3.1.0"
	}
}

Why not add a format key inside sops do a first pass with a map[string]interface{} and after that, by using the format do the second pass?

[autrilla]

@glerchundi no, the format is not always the same. YAML files look different, so do .env files.

[glerchundi]

@autrilla i thought the wrapping format was always the same, ignore me if it is not.

[jvehent]

Sops maintains the original format of the file, so we can't force a specific file structure on the user.

[mshkrebtan]

In case of YAML files like this:

:base_path: /var/gemstash
:cache_type: memory
:db_adapter: postgres
:db_url: postgres://gemstash:[email protected]/gemstash
:log_file: :stdout

I had to specify both --input-type and --output-type:

sops -d --input-type yaml --output-type yaml --verbose gemstash.yaml.enc

[bugcy013]

@mshkrebtan many thanks for giving workaround.