Add support for nested JSON

[ducaale]

See https://github.com/httpie/httpie/issues/78

[ducaale]

Some experiments I've done with nom https://gist.github.com/ducaale/5d3a3df08a39d128eedab362aac34850.

The plan is to support syntax like this:

$ xh httpbin.org/post foo.bar=5             # {foo: {bar: 5}}
$ xh httpbin.org/post foo[path.with.dots]=7 # {foo: {'path.with.dots': 7}}
$ xh httpbin.org/post foo[]=5 foo[]=6       # {foo: [5, 6]}
$ xh httpbin.org/post foo\.bar=5            # {'foo.bar': 5}

Things to consider:

1. Do we want to support bodies whose root is an array? e.g xh httpbin.org/post []=5 []=6
2. Do we want to support specifying an index when constructing array? e.g xh httpbin.org/post foo[1]=5 foo[0]=6
3. How object and array clashes should be handled? xh httpbin.org/post foo[]=5 foo[bar]=6

For the third question, https://github.com/jdp/jarg may have the answer:

$ jarg 'foo[]=5' 'foo[bar]=6'
{"foo": {"0": 5, "bar": 6}}

$ jarg 'foo[bar]=5' 'foo[]=6'
{"foo": {"bar": 5, "": 6}}

[blyxxyz]

1. Do we want to support bodies whose root is an array?

I don't see why not.

2. Do we want to support specifying an index when constructing array?

Are there use cases for that? It seems a bit niche, and treating anything between brackets as a string would be simpler.

3. How object and array clashes should be handled?

I think jarg's behavior is bad, because it changes the meaning of [] depending on the context. We could just forbid it, and if we find a good reason to handle it one way or the other it can be loosened later without breaking backward compatibility.

---

Question 4: How are these marked? foo.bar.baz=3 already has a meaning, changing that would be disruptive.

There's no fully backward compatible way to add a new separator. (It would be fully backward compatible to change the meaning of an existing separator using a flag, but that's tedious and doesn't let you mix.)

HTTPie already has :, ==, =, :=, @, =@ and :=@ as separators. Because of that, the characters =, @, :, ; (and \) are handled specially and may be escaped in keys and values.

Ideally a separator would be made up of those existing special characters. But @ is for files, and ; has to be escaped or it'll end the command, so that just leaves = and :. === and =: could work but I don't love them.

Another solution would be to leave the separator alone, and add a marker to the key. Maybe a leading dot? Then you'd have e.g. .foo.bar=5. That's reminiscent of jq, so hopefully it's not too weird. (Maybe inverting jq is a good strategy in general?)

That would also make the value more flexible, because then you can combine it with the existing separators.

It would still change the meaning of some commands that already work right now. I don't know how common keys with a leading dot are.

[ducaale]

Are there use cases for that? It seems a bit niche, and treating anything between brackets as a string would be simpler.

One use case I can think of is accessing the same index in array

$ xh httpbin.org/post [0].foo=7 [0].bar=5 [1].baz=3 # [{foo: 7, bar: 5}, {baz: 3}]

I think jarg's behavior is bad, because it changes the meaning of [] depending on the context.

I think empty strings is a valid use case although I am starting to think that optimizing for every edge case might not be ideal.

Another solution would be to leave the separator alone, and add a marker to the key. Maybe a leading dot? Then you'd have e.g. .foo.bar=5. That's reminiscent of jq, so hopefully it's not too weird. (Maybe inverting jq is a good strategy in general?)

I like this approach. However, I wonder if keys with dots are used that much. And if so, they could always be escaped, right?

[blyxxyz]

One use case I can think of is accessing the same index in array

Ah, of course. That makes sense.

I think empty strings is a valid use case

You could also express those with ../a trailing dot. A little weirder, but more predictable.

However, I wonder if keys with dots are used that much. And if so, they could always be escaped, right?

It's perhaps not that common, but I expect it does happen. Jakub brought it up as a constraint in the HTTPie issue, particularly for scripts (that may run without supervision).

Escaping is a bit problematic because it's not compatible with current versions. \. in a key is interpreted literally, the backslash stays, so there wouldn't be a way to write a command that works the same on both old and new versions. (That's also an issue with the leading dot.)

It would be nice to have a design that could also be adopted by HTTPie, I would be willing to port it. When we have a prototype (or a detailed design) we should post it in the issue for feedback.

[ducaale]

Some ideas to tackle the compatibility issue with the nested JSON syntax:

• Disable the feature when not in tty-mode.
• Let users opt-out of the feature via a flag. This means scripts will need to add this flag in addition to the --ignore-stdin flag.
• Make the feature opt-in via a flag or config.

[blyxxyz]

Disable the feature when not in tty-mode.

I think that would be very confusing.

Let users opt-out of the feature via a flag. This means scripts will need to add this flag in addition to the --ignore-stdin flag.

That wouldn't work, because old versions would reject the flag. And if you know your version is new enough you can just escape the dots. An environment variable could work, but it'd only be useful in limited cases. It'd still require users to take action to keep their old scripts working.

Make the feature opt-in via a flag or config.

A flag would be a bit unergonomic but besides that it would work. It would have perfect backward compatibility because trying to use it on an old version would simply give an error.

A config option would be confusing because it reduces copy/pastability. HTTPie also recommends against configuring default flags that change behavior significantly.

---

I just thought of another syntax with perfect backward compatibility: a leading colon. HTTPie and xh both currently parse :foo.bar=3 as a header with an empty name, and reject it.

The downside is that it's unintuitive. A leading dot at least looks like jq.

[ducaale]

A leading dot at least looks like jq

What if we use a single dot to activate the feature e.g xh httpbin.org/post . foo[]=5 foo[]=6?

[blyxxyz]

Hm, that one's interesting too.

But I'm not sure there's a reason to prefer it over a one-character flag.

[ducaale]

Httpie's approach to nested JSON syntax https://github.com/httpie/httpie/pull/1169

[ducaale]

HTTPie's nested JSON has been revised in a subsequent PR:

1. The syntax grammar is enforced now e.g unbalanced number of parenthesis is not allowed.
2. Numbers can be escaped in order to be treated as string e.g foo[\5]=hello.
3. Data type clashes will lead to a hard error e.g foo[x]=5 foo[]=6 or x=5 x[x]=7.
4. The "append by default" behavior has been abandoned e.g x=1 x=2 equals {"x": "1"} instead of {"x": ["1", "2"]}.

[ducaale]

Not super relevant, but a proposed JSON syntax for cURL has some form of nesting. See https://github.com/curl/curl/wiki/JSON

[tacomilkshake]

Excited for this capability in xh. Is using --raw the recommended approach until then?

[ducaale]

There are two other approaches besides --raw for sending an arbitrary request body:

• Pass data via redirected stdin e.g. echo '[1,2,3]' | xh httpbin.org/post (unfortunately, not documented at the moment).
• Read the request body from a file e.g. xh httpbin.org/post @file.json.

For more information, see https://httpie.io/docs/cli/raw-request-body.

[ducaale]

Resolved in #217