etcd perf: Write pages ASAP (optimistically)

[ptabor]

Write pages ASAP (optimistically)

It seems to me that bbolt starts 'write' of pages to disk when the transaction is committed: https://github.com/etcd-io/bbolt/blob/90fdc8d162431506d5fec5b199501ad86f5e15e6/tx.go#L537

Taking in consideration that:

• etcd does not rollbacks the transactions (as these are 'batch' commits)
• the page is really relevant if it's linked from the 'parents' page (or is a root page).
• etcd usually keeps appending to a 'last' page rather then doing random edits on set of pages

I think bbolt for etcd should:

• start writing leaf pages to disk as soon as they are full (even before commit was called).
• maybe even write `parents' and 'grant parents' of the leaf pages as soon as they modified.
• successful commit should:
  • write the free-pages index
  • write the 'root' page (and maybe a few (2-3) top level branch pages').
  • maybe 'flush'
• rolled back commit should return all the written pages back to the free pages list and do NOT modify the root page.

It should make the disk usage way more 'uniform' and commit locks substantially shorter.

Extracted from: https://github.com/etcd-io/etcd/issues/12924