tilt icon indicating copy to clipboard operation
tilt copied to clipboard

snapshot creation errors if it's larger than 4mb

Open nicks opened this issue 5 years ago • 11 comments

If you try to send a snapshot larger than 4mb, you hit the GRPC message size limit and get an error

nicks avatar Aug 11 '20 20:08 nicks

related to https://github.com/tilt-dev/tilt/issues/3194, which it seems was not fully fixed

nicks avatar Aug 11 '20 20:08 nicks

here's the error message on the tilt server side: grpc: received message larger than max (12096681 vs. 4194304)

nicks avatar Aug 11 '20 20:08 nicks

I think eventually we're just going to need to cap the number of logs in a snapshot, 4mb is a lot of logs for a local dev environment, so maybe that's a better place to start

nicks avatar Aug 11 '20 20:08 nicks

I think we may have started seeing this now that we're using snapshots again.

Error creating snapshot: posting snapshot failed. status: 500 Internal Server Error, response: error creating snapshot

andymartin-sch avatar Jan 07 '21 15:01 andymartin-sch

yep, the error i see on the backend is - grpc: received message larger than max (6775425 vs. 4194304)

nicks avatar Jan 07 '21 16:01 nicks

I'm not sure if it's the same underlying cause, but I got the same error this morning when creating a snapshot with Tilt v0.18.8: Error creating snapshot: posting snapshot failed. status: 500 Internal Server Error, response: error creating snapshot

andymartin-sch avatar Feb 04 '21 13:02 andymartin-sch

@milas I sent you a link to the datadog traces. Ya, I'm still seeing this hit limits grpc: received message larger than max (4921615 vs. 4194304) :\

nicks avatar Feb 04 '21 15:02 nicks

thanks @nicks !

andymartin-sch avatar Feb 04 '21 15:02 andymartin-sch

we got something like this recently in the form of:

  Snapshot: Error creating snapshot: posting snapshot failed. status: 413 Request Entity Too Large, response: snapshot too large. nodes over 1.0MB are: /view/logList/segments: 2.6MB 

andymartin-sch avatar Jun 10 '21 13:06 andymartin-sch

The 4 Mb limit seems less reasonable when considering that some users capture snapshots after running tilt ci. These snapshots would include all logs produced during a CI run, as opposed to brief logs created during a dev stint.

Any updates on this issue per chance? We'd love to be able to have a higher snapshot size limit, even if that were to cost $$.

sgreene570 avatar Feb 09 '22 21:02 sgreene570

We would also pay $$ if this were coupled with an improved onboarding workflow (https://github.com/tilt-dev/tilt/issues/3515)

andymartin-sch avatar Feb 10 '22 14:02 andymartin-sch