Content Negotiation & Accept headers with q-values

[vanHoesel]

A proposal to make writing better API servers with Dancer2...

As I recently have worked with Dancer2 and trying to use it as a REST or RESTful API-server, I encountered some troubles with content-negociation. My experience is that doing it right with Dancer is taking too much effort to get choose what do for each possible accepted content variant. This goes way beyond the existing serialisers or for the sake of completeness, beyond Dancer2::Serializer::Mutable. Not only does it need fixing to handle the different q-values but it also is not easy to make a difference between 'collection+json', 'application/json' or 'hal+json' and neither will it handle things like different file formats as 'image/jepg' or 'application/pdf' as being served as dynamic content. And beyond that, it only deals with HTTP-Accept MIME-types, not with language or encoding nor charset.

So this is what I'd like to see:

choose Accept (
    text_html
        => { # deal with HTML },
    application_json => {
        my %data = do_some_stuff_with( param('id') );
        return to_json %data;
    },
    'application/xml' => {
        my %data = do_some_stuff_with( param('id') );
        return to_xml %data;
    },
    ['image/jpeg', 'image/png']
        => { # make nice images },
    { %options }
);

what it should do, is execute the different code-REFs depending on what is handed through the HTTP-Accept header (an no other fancy wizzadry) and deal with the q-values.

Naturally, the Response HTTP Content-Type can be set by this mechanism, since you also know where to choose from.

This might look as more work when there is something like Mutable, but that does not give fine grain control over the tons off MIME-types there are. And then there is of course the list of options below.

The options could include:

• default => $MIME_type which could be used if the Accept header provides a */*
• Not-Acceptable => {…} to handle situations where there is no possible match, usually a Status: 406 can be treated as a good default, but returning Status: 300 and a list of links to alternates would be very nice, especially if that list can be generated from list of acceptable types.

This whole mechanism works for the other Accept headers as well and can be nested.

• choose Accept and
• choose Accept_Language
• choose Accept_Charset
• choose Accept_Encoding

Please... see this as a RFC, which I might implement myself with lot of your help and have some nice exercise in writing plugins, or you might be better off coding it with the team.

[vanHoesel]

if choose is going to be a problem, I would recommend HTTP_choose

or use different keywords, like:

• chooseAccept
• chooseAcceptLanguage
• chooseAcceptCharset
• chooseAcceptEncoding

[racke]

I acknowledge the idea, but I don't like the suggested implementation. Where would you put the code example above in your Dancer app?

[DavsX]

Isn't conditional matching the solution here? Right now it does not support those accept options, but it should not be that hard to implement/enable them.

[vanHoesel]

It is typical a piece of code that would fit in a route.

It is common to have a resource like: http://my_domain.tst/user/:idand to retrieve it's data like json (using Accept: application/json) or display a webpage (Accept: text/html), or generate a pdf. In all these cases the content needs to get 'retrieved' and either be parsed into json or through some template-engine. So, it looks quite obvious to place this part at the end of your code inside a single route.

I will rewrite the above example code to reflect what I just explained and elaborate a bit more on where things will go different:

get /user/:id => sub {

  my %user_data = retrieve_from_database( param('id') );

  choose Accept (
    'text/html' => sub {
      template 'user_profile_page' => { %user_data }
    },
    'application/json' => sub { return to_json \%user_data },
    'application/hal+json' => sub {
      return Data::HAL->new(
        resource => %user_data,
        links => [
          Data::HAL::Link
            ->new(relation => 'self', href => "/user/3456", title => "I'm here")
          Data::HAL::Link
            ->new(relation => 'dad',  href => "/user/1234", title => "John Doe")
          Data::HAL::Link
            ->new(relation => 'mum',  href => "/user/5678", title => "Jane Doe")
        ]
      )->to_json},
    'application/pdf' => sub {
      choose Accept_Language (
        'en_US' => sub { produce_imperial_pdf( %user_data ) },
        'en'    => sub { produce_metrical_pdf( %user_data ) },
        'nl'    => sub { produce_hollands_pdf( %user_data ) },
        { default => 'en' }
      )},
    { default => 'application/json' }
  );
}

So, basic user data will be retrieved form the database, which is common code for all the different Accept values. Additionally, to HAL+json, the links are being added. And lastly, if a PDF is requested, it allows the client to generate it in different versions, showcasing how to handle with nested choose commands.

[racke]

OK; but that is almost the same as if you would use if statements. Maybe it would be smarter to differentiate in a before hook or catchall route and pass it to separate routes (/user/123.pdf, /user/123.json, /user/123.html).

[vanHoesel]

get 'user/:id' => sub {

  my %user_data = retrieve_from_database( param('id') );

  choose Accept (
    'text/html' => sub {
      template 'user_profile_page' => { %user_data }
    },
    'application/json' => sub { return to_json \%user_data },
    'application/hal+json' => sub {
      return Data::HAL->new(
        resource => %user_data,
        links => [
          Data::HAL::Link
            ->new(relation => 'self', href => "/user/3456", title => "I'm here")
          Data::HAL::Link
            ->new(relation => 'dad',  href => "/user/1234", title => "John Doe")
          Data::HAL::Link
            ->new(relation => 'mum',  href => "/user/5678", title => "Jane Doe")
        ]
      )->to_json},
    'application/pdf' => sub {
      choose Accept_Language (
        'en_US' => sub { produce_imperial_pdf( %user_data ) },
        'en'    => sub { produce_metrical_pdf( %user_data ) },
        'nl'    => sub { produce_hollands_pdf( %user_data ) },
        { default => 'en',
          not_acceptable => sub { return "only English or Hollands available" }
        }
      )},
    [ 'image/jpeg', 'image/tiff', image/* ] => sub {
      if ( request->header('Accept') =~ m{image/jpeg} ) {
        return user_thumbnail_as_jpeg( param('id') )
      }
      elsif ( request->header('Accept') =~ m{image/tiff} ) {
        return user_portrait_large( param('id') )
      }
      else {
        return user_thumbnail( param('id') )
      }
    }
    { default => 'application/json' }
  );
}

Updated version, which does show some other nice feature in the 'spirit of PSGI': inside the subroutine to be executed, the request->header('Accept') will only return the value being chosen from.

It also does showcase the fact that it is not limited to one single MIME-type to select it's branch in the choose, but also it accepts an array_ref. In which case the order of the list is significant.

[vanHoesel]

Racke,

it is not an ordinary if statement, there is too much magic going on in the Accept-* headers, where the client can set preferences with the q-value. It will only choose the one that has the highest available option. If there are two equally highest ranked options, then the one that matches first inside the choose will be taken.

And there happens way more magic inside the choose, it also will 'modify' the header inside the sub-scope, as if there was only a single value in the header, where as outside, it will have the value of the client request.

What else happens is the the response header will be set accordingly to the matched value, no need to set that manually.

And it does provide in some useful options at the end, like default => $value and Not-Acceptable => \{…} which should produce a '406' status message or ideally a '300' with a list of valid options.

So, yes, as you suggest, it is all possible to write it in a hook. But then I have to write different hooks for each base route and differentiate it to more different routes.

I think my example code and idea behind it is that I do not changes or add the routes, for they are essentially the same resource, but only differ in their representation. Just like the original idea of `Dancer2::Serializer::Mutable'

Writing it in hooks, would als mean that for each variant there could be a sub-varient, as I demonstrated with the PDF... that would introduce yet another 3 entries in the routing table

[vanHoesel]

DavsX,

I had been pondering about 'conditional matching' and I could see the same technique being available as such, like:

get '/user/:id', {Accpet => 'application/HAL+json'} => sub {
  . . .
}

However there are a few things going wrong in that kind of approach... What if I also have:

get '/user/:id', {Accpet => 'application/pdf'} => sub {
  . . .
}

What would happen ?

Dancer will simply try to make a match of the route on a 'first match' basis. But the Accept header itself is actually the one that should be in control of which one to choose from.

The other thing is what will happen if there is no match available. Dancer would fall through and at the end generates a 404 Not Found which it should not, for the resource is there and could be found, but the content-negociation failed and so it must return a 406 Not Acceptable

[DavsX]

it does not matter, that you have multiple routes with the same path (only the accept header conditional being different). Yes, in that case the first route would match based on the path, but it would not get executed, because the conditional would not match. because of that, the first route would "pass" and the route matching algorith would continue on the second route.

It is possible to specify a route without the accept header conditional at the end, in which you could return a custom error (like the 406).

I have the feeling that all of this could be implemented nicely in a Dancer2 plugin; the right place to implement this kind of behaviour is maybe not in the core of Dancer2.

[racke]

I agree with @DavsX that this stuff should go into a plugin.

@vanHoesel: All in one route looks horrible to be honest. They are totally different things. I would rather have one module doing PDF, one doing HAL, etc.

[vanHoesel]

Again @DavsX , the order that the routes appear in the file is going to determine which will be executed, not the value of the Accept-* header... What if in the example below if would provide the header like Accept: application/pdf; q=0.7, text.html; q=0.1, application/json; q=0.6

get '/user/:id', {Accpet => 'application/json'} => sub {
  . . .
}

get '/user/:id', {Accpet => 'application/pdf'} => sub {
  . . .
}

the one that MUST be executed is the one with the highest q-value, in this case the PDF, not the one that first matches, not the JSON.

The other thing would be dealing with default and not-acceptable. Try to separate both at this moment but both have in common that it will use the 'fall-through' mechanism...

• If no Accept header is specified, then all the conditional-matchings will fail! Which one will you choose now? Ah… one of them that was mentioned before… sorry, too late
• If there is a Accept header, but not are matched, then again, all will fail! That is still fine, but now it depends on whether or not there was a /;0.0 in the header or not, in that case, it should report an error 406, otherwise if the q-value is not 0.0, then again, Dancer should had picked one of the options.

I remember while writing my first comment on conditional matching that I actually wanted to write down that I would advice against using it because of the nasty things going on with the q-values. The conditional matching only operates as a simple 'boolean filter'. The condition matches, or it does not.

[vanHoesel]

Something else that might not be so apparent is that my suggestion is in line with the concept of PSGI: inside the scope of the chosen subroutine, the Accept Header has it's modified value and also at the end of the subroutine the response header Content-Type has been set accordingly

[vanHoesel]

@racke, it might look globbered, so lets simplify the example:

get '/users/:id' => sub {

  do { some_stuff };
  my %user_data = retrieve_from_database( param('id') );
  do { some_more };

  choose Accept (
    'application/json' => sub { to_json \%user_data },
    'text/html'        => sub { template 'user_profile_page' => { %user_data } },
    'audio/mpeg3'      => sub { text_to_speech( %user_data ) },
    { default => 'text/html' }
  );
};

Doesn't it look nice? and I deliberately used a MIME-type that is not being handled with one of the standard serialisers.

And then as a counter example of what you say that 'those' are different and how Dancer2 does give examples of how they suggest on using splat, which is btw a very unreliable thing to do:

get 'some_path_to/*.*' => sub {
  my ($filename, $type) = splat;
  . . .
}

this implies that there are different representations of this same resource some_path_to/filename, somehow falsely recognised now by $type.

From the point of view of an API developer: whatever is requested defined by the Accept header is totally irrelevant, it is the same thing, only with different representation. When writing REST API's you define one single route for a resource, no matter how it is being represented, that is only the responsibility in the last stage at delivery time, the return statement.

[vanHoesel]

And in general: no it does not need to be 'core', it can be implemented as one or several plugins

[rleir]

Agreed that this is best done in a plugin. Maybe a route decorator plugin.

This issue is stale. Close it? When you get a moment, thanks!

[vanHoesel]

it is not 'stale' during the Londen Perl Workshop i will tel more about it. There is one nasty problem with the plugin I am developing, the underlying HTTP::Headers::ActionPack module by Steven Little and Dave Rolsky is not RFC compliant - it gets quite OKish but it made me wonder if I would use faulty modules to implement the plugin, send in a bug report - or just write my own ContentNegotiation plugin

[SysPete]

I think @rleir's view is not that your requirements and ideas are stale but that since there has been no update for > 12 months and agreement seemed to have been reached that a plugin is the correct route then this ticket should be closed.

@vanHoesel do you have a public repo for your new plugin? Perhaps you could add a link in a comment to this issue so these things stay connected and then close the issue?

[vanHoesel]

Okay - it needed to be done anyways - but after writing hours on POD, this is what the module more or less would look like: HTTP::ContentNegotiation. Hopefully the Synopsis is clear enough.

[rleir]

@vanHoesel Very nice. Will it be more than an example? I will use get_language most likely. Then, with HTTP 'Accept-Language' and the browser's locale setting, we can be multilingual (is this the best approach?) @SysPete, yes, exactly. Thanks -- Rick

[vanHoesel]

@rleir I can not say if it is the best approach. Although I wrote this with REST-api's in mind, HTTP::ContentNegotiation is not limited to api's. It is certainly 'a way' to serve a web-page or an api-response in a language defined by the browser or clients preferences. More difficult it might become when you want the end user to give the ability to select a language. You would had to manually forge the HTTP Request to include a different Accept-Language . . but certainly for first-time-visitors, the Home page could be served in the language of the browser.

[veryrusty]

@vanHoesel++ that looks nice! I'll make further suggestions over on your repo!