Ensure router shows up in API reference

[ktrott]

[mik01aj]

:+1: The router docs are missing :(

[mik01aj]

There is something in http://netflix.github.io/falcor/documentation/router.html#walkthrough-building-a-router-for-netflix-like-application, but nothing about router in the API docs.

[jhusain]

Should get some API docs on Router soon, certainly before we come out of developer preview.

[mik01aj]

I'll make some notes :) As I dig through the introduction I figured out that:

• the route definition must have route, which is required, and a get or set handler (or both).
• a route handler can return a list of PathValues, or a JSONGraphEnvelope, or maybe something else too.
• a route handler can be called many times for the same route (https://github.com/Netflix/falcor/issues/643)
• a route handler can return more PathValues than asked for; Falcor will ignore whatever is not needed for the request. This can be (ab)used to pre-fill some data of the referenced objects to avoid unnecessary requests to backend services.

[jhusain]

Much appreciated.

[mik01aj]

Btw... Making a list of PathValues in each route is a lot of hassle. Why don't you provide an utility to wrap a JsonGraph in a JsonGraphEnvelope with all the paths? Imho specifying the paths all the time is a lot of redundancy.

[mik01aj]

a route handler can return more PathValues than asked for; Falcor will ignore whatever is not needed for the request. This can be (ab)used to pre-fill some data of the referenced objects to avoid unnecessary requests to backend services.

I noticed that this is actually not totally true; Falcor Router does not ignore anything I return in my handler. Everything I return in route handler is passed on the wire and then non-matching stuff is ignored only in the client-side Model.

EDIT: Actually, I don't understand the reasoning behind this behaviour. If the data returned from route was filtered to match the requested PathSet, the code for all the routes implemented by all the Falcor users could be greatly simplified because they could just always return everything they got from the backend. (All for a little performance penalty. Plus returning anything non-declared in route handler wouldn't work, thus breaking the "optimization" I pointed out above.).

[ThePrimeagen]

@mik01aj you are correct. The model itself does not merge in anything except for what is requested. So its definitely a mistake there.

[mik01aj]

I looked at the DataSource docs and thought, the routes in router look pretty much like a DataSource. Correct?

P.S. I edited my comment above.

[ThePrimeagen]

@mik01aj Yes a router is a specialized ("implementor") DataSource.

[mik01aj]

The doc page says:

In addition to returning either JSON Graph envelopes or PathValues synchronously, Router handlers can also return their data asynchronously (...)

I tried returning both a JsonGraph and a JsonGraphEnvelope and it didn't work. Maybe the docs are outdated?

[ThePrimeagen]

https://github.com/Netflix/falcor-router/blob/master/test/unit/core/get.spec.js#L439 https://github.com/Netflix/falcor-router/blob/master/test/unit/core/get.spec.js#L474

Example of returning jsonGraph.

[mik01aj]

Ah, I see. When I return {jsonGraph: stuff} it works. With {jsonGraph: stuff, paths: pathSet} it doesn't.

So now I have generic filtering for all the routes with just one simple function:

function makeFilteredJsonGraph(pathSet, jsonData) {
    let model = new falcor.Model({cache: jsonData});
    return {jsonGraph: model.getCache(pathSet)};
}

[ThePrimeagen]

That cannot be it either: https://github.com/Netflix/falcor-router/blob/master/test/unit/core/get.spec.js#L35

I know I have unit tests that test path passing is ok.

[ThePrimeagen]

If you don't pass back paths, the requested pathSet will be used: https://github.com/Netflix/falcor-router/blob/master/src/run/conversion/noteToJsongOrPV.js#L52

[mik01aj]

@michaelbpaulson good point. So what I undestand now is that I can return either an Array of PathValues or a JsonGraph, and if I return a JsonGraph without specified paths, then it will be filtered to match the pathSet that was originally passed to the route handler. On the other hand, if I return an Array of PathValues, they are not filtered in any way and the result will be passed through HTTP as it is. This is pretty confusing and inconsistent.

Btw. this also implies there is no point in having my own filtering function as I posted above.

[mik01aj]

In short, all this also means that if the route handler is returning anything more than it was declared (for example, because of lack of expressive power in the route strings), the additional values may make it or not to the client.

Let say, a route for topics.{integers}.['name', 'firstEntry'] returns a JsonGraph like this:

{jsonGraph: {
  topics: {
    [topicId]: {
      name: 'hello',
      firstEntry: {$type: 'ref', value: ['entries', entryId]},
    }
  },
  entries: {
    [entryId]: {text: 'hello world!'}
  },
}}

Then we have following problems:

• if the client requests ['topics', 123, 'firstEntry', 'text'] and the route handler returns a result as above, the entries[entryId].text will be stripped from JSON graph and will not make it to the response (Falcor will try to find another route to get it).
• if the client requests ['topics', 123, 'name'] and the route handler returns a result as above, but as PathValues, the entry text will be passed to the client, in spite of being not requested.
• Note that I can't declare these both data points in one route. :(

[ThePrimeagen]

@mik01aj

• You can accomplish the same thing with jsonGraph as pathValues. Extra information can be sent, though no one will hear it.

Given your above example

{jsonGraph: {
  topics: {
    [topicId]: {
      name: 'hello',
      firstEntry: {$type: 'ref', value: ['entries', entryId]},
    }
  },
  entries: {
    [entryId]: {text: 'hello world!'}
  },
}}

And the request is ['topics', 1234, 'firstEntry', 'text'] then this won't be stripped. The only case that this will be stripped is when bad route creation happens. If you had a route that matched ['topics', 1234, 'firstEntry'] but returned that extra data, through jsonGraph, without specifying a path, of course that data will be dropped before sending across the wire! But if pathValues or jsonGraph with paths key is used then the data will be dropped at the Model. Either way, the data will be lost.

I am getting confused. Are you seeing buggy behavior or are you attempting to do naughty things?

[mik01aj]

I'd call this inconsistent behaviour.

In my point 1, I as for data that can be delivered by one route, but Falcor will fire several routes. In point 2, there will be some extra data sent over the wire, and stripped only in the client-side Model.

Maybe you'd say that I'm doing a wrong thing returning entries[entryId].text from a route that doesn't declare this. But if so, why doesn't Falcor always stripe it, or throw an error? The current behaviour is weird.

[ThePrimeagen]

I think that you are drawing conclusions that are incorrect. Its doing exactly what you told it to do. If you return a PathValue, its merged into the cache based on the path thats provided. If you return JSONGraph with paths it merges the jsonGraph with the paths provided. If you return JSONGraph without paths, then you are telling the router, use the incoming set of paths to the route.

Not maybe, I am saying you are doing something wrong by returning more data than your route specified.

If you want to return references, as above, you should be specify a couple routes.

{
    route: 'topics[{integers}].firstEntry',
    get: function(pathSet) {
        return someCallToGetEntries(pathSet[1]).map((entry, index) => {
            return {
                path: ['topics', index, 'firstEntry'],
                value: Model.ref(['entries', entry.id])
            };
        });
    }
}, {
    route: 'entries[{integers}].text',
    get: function(pathSet) {
        return getEntryById(pathSet[1]).map((entry) => {
            return {
                path: ['entries', entry.id, 'text'],
                value: entries.text
            };
        });
    }
}

Then I would query for ['topics', 1234, 'firstEntry', 'text'], hitting both routes, and get only the information that was asked for.

Though the router is not super efficient as of now, its our goal to write the most efficient code possible. Which means we need implementors to behave or else I need to create the intersection from requested pathSets (arrays) and outgoing jsonGraph (tree), which is not cheap.

@mik01aj could you please give me a real scenario why you would emit extra information (I truly am not trying to sound like an ass or patronize you. I mean this)? If there is a real use case, lets work together and build the correct solution. If this is just trying to add extra user safety measures at the cost of performance, I am going to kick against the goads.

BTW, though the router has not had its performance improvements, all changes are thought in performance cost. Its very important. With 69.1 (last publicly known member count, 15Q3) million members, adding performance hits cost real money.

[mik01aj]

@mik01aj could you please give me a real scenario why you would emit extra information (...)? If there is a real use case, lets work together and build the correct solution.

Yep... maybe this whole story would be clearer if I explained this. The reason is that my Falcor routes call endpoints of a REST-ish API, and that API is not strictly REST. For example, when I ask for /topics/123 I get the topic info, plus the first entry data. Of course I could just use the entry id and return a reference as you suggest (while ignoring all other entry data), but this wouldn't be very efficient, because then the entry route would have to make one more request to /entries/456 to get the entry data. I'd like to take advantage of the data that I already got from my API by prefilling the cache with that data to avoid extra requests when I don't need them.

---

I'll give an example.

I would expect the router to work like this for given pathSets:

1. Look at the pathSets and find a matching route for each of them.
2. Cut each pathSet to match what the matched route declared.
3. If some routes are matched for many pathSets, merge the pathSets into one (this should be always possible) so that no route is called more than once.
4. Fire the routes for their pathSets in parallel. (Promise.all or similar)
5. Merge all the results into an JsonGraph object (without any filtering).
6. Go through the resulting JsonGraph: if some point that was requested from route handlers is missing, put an empty Atom in this place. (This could also possibly trigger a warning for developers.)
7. For all references in the resulting JsonGraph, use them to resolve the missing pathSets.
8. Make a list of pathSets that are still needed (needed = references point to them and they are not yet in JsonGraph)
9. If that list is not empty, use that list to start again from step 1.
10. Filter the resulting JsonGraph to contain only the data requested in the original pathSets.
11. Return the JsonGraph.

Let say I have 2 routes: topics[{integers}].['firstEntry','title'] and entries[{integers}].text.

So for example, for input pathSets [['topics', 123, 'firstEntry', 'text']] this would be:

1. matched route topics[{integers}].['firstEntry','title']

2. pathSet for route: ['topics', 123, 'firstEntry'] (textcut off)

5. so we got:

   {jsonGraph: {
   topics: {
     123: {
       name: 'hello',
       firstEntry: {$type: 'ref', value: ['entries', 456]},
     }
   },
   entries: {
     456: {text: 'hello world!'}
   },
   }}
   

6. Nothing is missing

7. So we wanted [['topics', 123, 'firstEntry', 'text']], asked for ['topics', 123, 'firstEntry'] and got {$type: 'ref', value: ['entries', 456]} in that place, so the new pathSet is ['entries', 456, 'text']...

8. ...and it is already present in the JsonGraph. Nothing more is needed.

10. Removing topics.123.name, it's not needed

11. done

---

Right now I see that the filtering is actually up to the user. I tried to find a generic solution so I don't have to implement it manually for every route handler, but then I lose some of the flexibility (so I can't return more than asked for). The algorithm I presented would be perfect (imho), because it gives me this flexibility while also letting me return as much as I have in my route handlers and not to worry about filtering each time.