Ian O'Connell

[Api Discussion via docs] Adding a more manual/advanced scalapb rule

8

In the change i made to make these rules aspect based previously it has blocked/made difficult some of scalapb's options that don't translate well into aspects (and aren't supported in...

Now we really just use Bijections or Injections, so having Rep's too just leads to confusion. We should just remove all instances

Adds a pass at an alternative lazy parquet reader implementation

2

Needs some more work before merging around cleaning it up, more benchmarks and seeing what does/doesn't need to be in lui. TBD are real world tests against jobs with manual...

Scala implicit ordering fun -- have 2 forms of sketch, defaulting to …

5

…the existing @johnynek as we were discussing for the sketch method. More of a consideration than hard PR

Scalding repl if just override the entry point will fail

3

https://github.com/twitter/scalding/blob/develop/scalding-repl/src/main/scala/com/twitter/scalding/ReplImplicits.scala#L139 Refers to the ScaldingShell object, if a user makes their own shell inheriting from ScaldingShell this will break how the code is built I believe. (I thinnk @ Twitter...

Blocked on a new version of addSbtPlugin("io.spray" % "sbt-boilerplate" % "0.5.0") https://github.com/sbt/sbt-boilerplate/issues/8

ManagementFactory.getMemoryMXBean() The downside of using runtime.freeMemory is that it will only decrease once a GC has gone through. So it will underestimate the actual free memory, possibly flushing more often...

Things like the base bolt, and other classes should all be marked as private to separate classes we wish to provide as a stable interface

We should strip the callstack printing from the timeout exception

1

https://github.com/twitter/summingbird/blob/f64ec6b8de54242c6be712502d315d9dd99e678a/summingbird-online/src/main/scala/com/twitter/summingbird/online/executor/AsyncBase.scala#L95 It doesn't add anything to expose an internal callstack there. Also: We should be bubbling this up to the reportError function of the base bolt -- right now this...

In online planning like name nodes we should strip the identity keyed producers

1

We've had strange error cases around them not being expected. It would be good to just strip the graph to just operationally require components