appengine-pipelines When the pipeline info exceeds 32MB the UI fails

Because the UI attempts to load all the information needed to display the outline in a single http request, and AppEngine caps data returned to 32MB if the info required to display the outline exceeds this, it results in: "Unable to parse JSON" being displayed, and showing nothing.

Oct 17 '14 22:10 tkaitchuck

https://github.com/Khan/appengine-mapreduce has fixed the problem in their implementation by providing truncated values.

I've implemented a fix for this as well, and the only catch was to ensure that we aren't truncating values for urlsafe Keys (otherwise pipeline children won't load if they have keys > 100 char truncation limit).

Oct 17 '14 22:10 soundofjw

Hey @tkaitchuck do you have a contact at Khan that might be interested in upstreaming their fixes to this repo?

Oct 19 '14 18:10 AngryBrock

I've started taking a look at bringing the truncation & &depth= UI features in.

Kahn took their own approach to the consistency problem resolved here with _BarrierIndex (see their _BarrierListener).

I've begun to sync up these changes and I will create a pull request when I wrap it up.

Nov 16 '14 17:11 soundofjw