September 14 2018
HTTP content negotiation is a mechanism by which web servers consider request headers in addition to the the URL when determining which content to include in the response. A common use cases for this is response body compression, wherein a server may decide to gzip the content if the request arrived with an Accept-Encoding: gzip
header.
Support for content negotiation in HTTP servers is a mixed bag. Apache provides good built-in support for this. NGINX does not offer anything comparable although a rough approximation is possible via configuration directives. Unfortunately I can't find documentation on IIS. AWS S3 static website hosting, which is used to serve this blog, provides no facility for this whatsoever.
Over the past few years, CDNs have evolved to help address this problem in a few days.
Most CDNs can compress content on the fly, even if the origin only serves uncompressed. Support for gzip is de rigueur, with CloudFront supporting Brotli as well. In practice, however, this can be limited. For example, AWS CloudFront won't compress anything under 1KB or over 10MB. In addition, compression is typically more effective the more CPU you spend on it though this effect is non-linear. For example, running gzip at level 9 can produce content that is 10s of percent smaller than level 1, but requires several times the processing power. As a result, CDNs are typically configured to run at fairly low optimization levels.
Recently CDNs have also begun to allow applications to run business logic at the edge. CloudFlare workers, AWS Lambda@Edge and Fastly VCL are all examples of this.
Felice Geracitano had the clever idea to use Lambda@Edge on AWS CloudFront to implement a bare bones content negotiation scheme for the purpose of supporting Brotli. While there are some issues with his implementation, the concept of performing content negotiation in JavaScript on the CDN and using the result to drive fetching a different resource from the origin is a powerful one.
Accept-Encoding: gzip, br;q=0.9
indicates that if content encodings for both gzip
and br
are available, it prefers gzip
. Vary
header to ensure that downstream caches are not confused by the content negotiation process.Below is an implementation of this for AWS CloudFront, and is being used to handle traffic to https://blog.std.in. The code is MIT licensed and is derived from pgriess/http-content-negotiation-js.
First, the http-content-negotiation
module:
/*
MIT License
Copyright (c) 2018 Peter Griess
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
/*
* Given an array of AWS Lambda header objects for headers that support
* ','-delimited list syntax, return a single array containing the values from
* all of these lists.
*
* Assumptions
*
* - HTTP headers arrive as an array of objects, each with a 'key' and 'value'
* property. We ignore the 'key' property as we assume the caller has supplied
* an array where these do not differ except by case.
*
* - The header objects specified have values which conform to section 7 of RFC
* 7230. For eample, Accept, Accept-Encoding support this. User-Agent does not.
*/
const splitHeaders = function(headers) {
return headers.map(function(ho) { return ho['value']; })
.reduce(
function(acc, val) {
return acc.concat(val.replace(/ +/g, '').split(','));
},
[]);
};
/*
* Parse an HTTP header value with optional attributes, returning a tuple of
* (value name, attributes dictionary).
*
* For example 'foo;a=1;b=2' would return ['foo', {'a': 1, 'b': 2}].
*/
const parseHeaderValue = function(v) {
const s = v.split(';');
if (s.length == 1) {
return [v, {}];
}
const attrs = {};
s.forEach(function(av, idx) {
if (idx === 0) {
return;
}
const kvp = av.split('=', 2)
attrs[kvp[0]] = kvp[1];
});
return [s[0], attrs];
};
/*
* Given an array of (value name, attribute dictionary) tuples, return a sorted
* array of (value name, q-value) tuples, ordered by the value of the 'q' attribute.
*
* If multiple instances of the same value are found, the last instance will
* override attributes of the earlier values. If no 'q' attribute is specified,
* a default value of 1 is assumed.
*
* For example given the below header values, the output of this function will
* be [['b', 3], ['a', 2]].
*
* [['a', {'q': '5'}], ['a', {'q': '2'}], ['b', {'q': '3'}]]
*/
const sortHeadersByQValue = function(headerValues) {
/* Parse q attributes, ensuring that all to 1 */
var headerValuesWithQValues = headerValues.map(function(vt) {
var vn = vt[0];
var va = vt[1];
if ('q' in va) {
return [vn, parseFloat(va['q'])];
} else {
return [vn, 1];
}
});
/* Filter out duplicates by name, preserving the last seen */
var seen = {};
const filteredValues = headerValuesWithQValues.reverse().filter(function(vt) {
const vn = vt[0];
if (vn in seen) {
return false;
}
seen[vn] = true;
return true;
});
/* Sort by values with highest 'q' attribute */
return filteredValues.sort(function(a, b) { return b[1] - a[1]; });
};
/*
* Perform content negotiation.
*
* Given sorted arrays of supported (value name, q-value) tuples, select a
* value that is mutuaully acceptable. Returns null is nothing could be found.
*/
const performNegotiation = function(clientValues, serverValues) {
var scores = [];
for (var i = 0; i < clientValues.length; ++i) {
const cv = clientValues[i];
const sv = serverValues.find(function(sv) { return sv[0] === cv[0]; });
if (sv === undefined) {
continue;
}
scores.push([cv[0], cv[1] * sv[1]]);
}
if (scores.length === 0) {
return null;
}
return scores.sort(function(a, b) { return b[1] - a[1]; })[0][0];
};
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const headers = request.headers;
if ('accept-encoding' in headers &&
!request.uri.startsWith('/gzip/') &&
!request.uri.startsWith('/br/')) {
const SERVER_WEIGHTS = [
['br', 1],
['gzip', 0.9],
['identity', 0.1],
];
const sh = splitHeaders(headers['accept-encoding']);
const ph = sh.map(parseHeaderValue);
const qh = sortHeadersByQValue(ph);
const rep = performNegotiation(qh, SERVER_WEIGHTS);
if (rep && rep !== 'identity') {
request.uri = '/' + rep + request.uri;
}
}
callback(null, request);
};
... and now the request handler:
/*
MIT License
Copyright (c) 2018 Peter Griess
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
exports.handler = (event, context, callback) => {
const response = event.Records[0].cf.response;
const headers = response.headers;
headers['Vary'] = [{key: 'Vary', value: 'Accept-Encoding'}];
callback(null, response);
};
The origin for https://blog.std.in/ is an S3 bucket configured for static website hosting. There are 3 different versions of each piece of content -- one un-processed, one compressed with gzip, and another compressed with Brotli. The compressed content lives in a shadow directory hierarchy under /gzip
and /br
respectively, allowing the path for compressed content to be computed by prepending the requisite directory.
There are two handlers -- an origin request handler and an origin response handler. There are no viewer handlers, allowing CloudFront to skip this logic entirely when serving a cache hit. The origin request handler performs the content negotiation, parsing the Accept-Encoding
header and comparing its requirements with what's provided by the S3 bucket serving as the origin. It selects the best match and updates the URI to fetch from the origin. The origin response handler sets a Vary: Accept-Encoding
header on the response indicating that content was negotiated based on the value of the Accept-Encoding
header. The resulting response is then cached in CloudFront.
Finally, the CloudFront distribution is configured with the "Cache Based on Selected Request Headers" setting set to include Accept-Encoding
. This has the effect of CloudFront incorporating the browser's Accept-Encoding
header in its cache key when looking up a response. In addition, this prevents CloudFront from stripping the browser's Accept-Encoding
header before the origin request handler has a chance to execute.
It is surprising to me that the Vary
header is not being added automatically by CloudFront as enabling "Cache Based on Selected Request Headers" and adding Accept-Encoding
explicitly indicates that the content for the given URL may vary by the value of this header. This seems like a pretty clear indication that CloudFront should be adding a Vary: Accept-Encoding
header to the response automatically.