Proposal: Filters and big data API

[davisjam]

This is a proposal, not a bug report. As such it's a bit unfocused, mostly brainstorming.

• Filter the package.jsons from meta/
• Filter the .tgzs from current/, optionally based on the filtered package.json's from meta/

This is similar to the code search and AST execute APIs, but seems more general-purpose.

I know I can grep through the files in pool/ for this, but that's brittle because it depends on Gzemnid internals. Can Gzemnid expose this facility?

Also, Gzemnid queries are a big data problem. Thoughts on exposing some kind of producer-consumer web API so I can easily distribute queries across a cluster of workers?

I'm happy to work on a PR if we can decide on the API (e.g. celery or similar).