record literals with quiet errors should drop them

[mccanne]

This should result in an empty record but doesn't:

echo 'error("quiet")' | zq 'yield {foo:this}' -

[Cybergate9]

just a FYI - in case others come across same problem - my current work-arounds for this until fixed:

1st go was:

zq -Z 'over this 
| yield {timestamp:timestamp, 
time:cast(timestamp*1000000000, <time>), 
post:data[0].post, 
url:(is_error(attachments[0].data[0].external_context.url) ? "none" : attachments[0].data[0].external_context.url), 
img:(is_error(attachments[0].data[0].media) ? "none" : attachments[0].data[0].media)}  
|  collect(this)  
|  yield this' your_posts_2.json

this 'conditionals' empty source fields (i.e. attachments[0].data[0].external_context.url) into "none" strings in the output fields if they're empty or absent. Barely better than 'missing', in fact worse in a structured sense I guess, but otherwise we get "url":{"error":"missing"} type output which is distinctly non 'string'y and will be painful downstream..

2nd go: I think below is the actual work-around to be used, much more zed'y and elegant, just add a 'cut with quiet()s', and the result is empty or absent field in input, yields no field at all in output. Depending on downstream processing depends on whether method 1 or 2 is better in your use case. Not great one has to (re)name all the fields in cut but ce le vie..

zq -Z 'over this 
| yield {timestamp:timestamp, 
time:cast(timestamp*1000000000, <time>), 
post:data[0].post, 
url:attachments[0].data[0].external_context.url, 
img:attachments[0].data[0].media} 
| cut timestamp, time, post, quiet(url), quiet(img) 
| collect(this) 
|  yield this' your_posts_2.json 

for interest the "your_posts_2.json" is directly from a FB json export.

[Cybergate9]

even better.. See previous post - we all know having to name fields twice in a script is going to lead to problems later, so I kept scratching the itch..

solution - just use 'cut' instead of 'yield', because 'cut' construct honors 'quiet', won't work in all use cases but does in mine..

over this 
| cut timestamp:=timestamp,
datetime:=cast(timestamp*1000000000, <time>),
post:=data[0].post,
url:=quiet(attachments[0].data[0].external_context.url),
img:=quiet(attachments[0].data[0].media)
| collect(this) 
| yield this

[philrz]

This came up again in a response to a question from a community user (Slack). Repros are with Zed commit 2eaa6d9.

Their question:

Hi, I am using the ip command on linux to produce network interface info, and formatting the output using zq. I have it kind of working now: ip -j a | zq -f csv 'over this | {ifname,address,ip_addr:addr_info[0].local} | fuse' - Just wondering what's the easiest way to get rid of all the ((string,error(string))) and all (or most of) the quotes in the output, and maybe additionally simplify all the error(""missing"") to simply an empty string or something short. I could pipe the result to sed or perl to get the desired result using regex. But it would be nice and instructive to do most of it using simple syntax in zq. Thanks

Here's an example repro from my own Linux VM where interface enp0s8 lacks an IP address.

$ ip -j a
[{"ifindex":1,"ifname":"lo","flags":["LOOPBACK","UP","LOWER_UP"],"mtu":65536,"qdisc":"noqueue","operstate":"UNKNOWN","group":"default","txqlen":1000,"link_type":"loopback","address":"00:00:00:00:00:00","broadcast":"00:00:00:00:00:00","addr_info":[{"family":"inet","local":"127.0.0.1","prefixlen":8,"scope":"host","label":"lo","valid_life_time":4294967295,"preferred_life_time":4294967295},{"family":"inet6","local":"::1","prefixlen":128,"scope":"host","valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":2,"ifname":"enp0s3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"fq_codel","operstate":"UP","group":"default","txqlen":1000,"link_type":"ether","address":"08:00:27:61:d5:c7","broadcast":"ff:ff:ff:ff:ff:ff","addr_info":[{"family":"inet","local":"10.0.2.15","prefixlen":24,"broadcast":"10.0.2.255","scope":"global","dynamic":true,"noprefixroute":true,"label":"enp0s3","valid_life_time":82283,"preferred_life_time":82283},{"family":"inet6","local":"fe80::d24d:fcdb:fffc:f57f","prefixlen":64,"scope":"link","noprefixroute":true,"valid_life_time":4294967295,"preferred_life_time":4294967295}]},{"ifindex":3,"ifname":"enp0s8","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"fq_codel","operstate":"DOWN","group":"default","txqlen":1000,"link_type":"ether","address":"08:00:27:c9:4f:25","broadcast":"ff:ff:ff:ff:ff:ff","addr_info":[]}]

$ zq -version
Version: v1.2.0-51-g2eaa6d90

$ ip -j a | zq -f csv 'over this | {ifname,address,ip_addr:addr_info[0].local} | fuse' -
ifname,address,ip_addr
lo,00:00:00:00:00:00,"""127.0.0.1""((string,error(string)))"
enp0s3,08:00:27:61:d5:c7,"""10.0.2.15""((string,error(string)))"
enp0s8,08:00:27:c9:4f:25,"error(""missing"")((string,error(string)))"

Like the workaround shown by @Cybergate9 above, @nwt ended up showing them how to use a conditional to to replace the "missing" errors with an empty string. Indeed, this does the trick.

$ ip -j a | zq -f csv 'over this | {ifname,address,ip_addr:missing(addr_info[0].local)?"":addr_info[0].local} | fuse' -
ifname,address,ip_addr
lo,00:00:00:00:00:00,127.0.0.1
enp0s3,08:00:27:61:d5:c7,10.0.2.15
enp0s8,08:00:27:c9:4f:25,

However, I was left wondering why quiet() didn't do the trick. It's easier to see why if we output as ZSON and drop the fuse.

$ ip -j a | zq -Z 'over this | {ifname,address,ip_addr:quiet(addr_info[0].local)}' -
{
    ifname: "lo",
    address: "00:00:00:00:00:00",
    ip_addr: "127.0.0.1"
}
{
    ifname: "enp0s3",
    address: "08:00:27:61:d5:c7",
    ip_addr: "10.0.2.15"
}
{
    ifname: "enp0s8",
    address: "08:00:27:c9:4f:25",
    ip_addr: error("quiet")
}

As long as the record literal sees error("quiet") as a value of type error, fuse assigns a union type to ip_addr and that has knock-on effects in the CSV output.

$ ip -j a | zq -Z 'over this | {ifname,address,ip_addr:quiet(addr_info[0].local)} | fuse' -
{
    ifname: "lo",
    address: "00:00:00:00:00:00",
    ip_addr: "127.0.0.1" ((string,error(string)))
}
{
    ifname: "enp0s3",
    address: "08:00:27:61:d5:c7",
    ip_addr: "10.0.2.15" ((string,error(string)))
}
{
    ifname: "enp0s8",
    address: "08:00:27:c9:4f:25",
    ip_addr: error("quiet") ((string,error(string)))
}

$ ip -j a | zq -f csv 'over this | {ifname,address,ip_addr:quiet(addr_info[0].local)} | fuse' -
ifname,address,ip_addr
lo,00:00:00:00:00:00,"""127.0.0.1""((string,error(string)))"
enp0s3,08:00:27:61:d5:c7,"""10.0.2.15""((string,error(string)))"
enp0s8,08:00:27:c9:4f:25,"error(""quiet"")((string,error(string)))"

Conclusion: If the the "quiet" error had been dropped in the record literal as this issue proposes, the user would have been able to use quiet() for a simpler solution.