VIP Go file service and Photon JS

[scottsweb]

Right now we are using Photon hooks to re-write images to use the VIP Go file service (https://github.com/Automattic/vip-jetpack/blob/master/jetpack-misc.php#L22). The hope was the Photon JavaScript for detecting device resolution and scale would continue to work and provide the correct image query parameters. Unfortunately the JS has a number of checks that fail.

The two Photon JS files used for modifying images are:

https://s0.wp.com/wp-content/js/devicepx-jetpack.js?ver=201607 (which contains a number of URL checks e.g. img.src.match(/^https?:\/\/([^\/]+)\.files\.wordpress\.com\/.+[?&][wh]=/)).

and

https://tumbleweed.simonwheatley.co.uk/wp-content/mu-plugins/jetpack/modules/photon/photon.js?m=1432675342g (which may be completely unnecessary).

Our solution to this should be to completely disable Photon and to re-write devicepx-jetpack.js to correctly support the VIP Go file service. This may require us to pass in the current URL into the JavaScript (via variable in the head or a data-attribute) to ensure we are only modifying images served from the current site domain.

Whilst the VIP Go file service only supports one single query parameter we may also be able to prioritise the order in which the query parameters are added to ensure we are giving the best image we can. In future we will be able support multiple query parameters.

[simonwheatley]

I think we should investigate this issue to see how simple it is to fix, and judge whether to keep it in the Minimum Supportable Product on that basis.

[nickdaugherty]

I think we should help improve JP by making it more host agnostic.

In an ideal scenario, we use JP Photon straight away, with a small filter to set the photon domain (like what we have). That will reduce the differences between prod and dev, assuming the Go Files Service behaves the same as Photon of course :)

[ethitter]

We could add a clause to scaleImage in devicepx-jetpack.js that checks for a variable from wp_localize_script() with the filtered domain name: https://wpcom.trac.automattic.com/browser/trunk/wp-content/js/devicepx-jetpack.js?rev=127396#L428. Even though the hosted file doesn't change often, forking it doesn't help anyone else who's trying to host Photon themselves (if anyone even is).

I don't see anything in https://github.com/Automattic/jetpack/blob/d62e795fb37336d971a066e9735d823ab200fb56/modules/photon/photon.js that would need updating, but the same approach would work if necessary.

[m]

Why not just put the images through photon normally?

[darthhexx]

As a backdrop; the Go Files service is a distributed file storage system into which the image processing features have been integrated, whereas Photon processes images requested from a remote origin.

Key points for the use of the Go Files services:

• Photon allows cache invalidation using the exact query string and Accept header matches, whereas Go Files allows fine grained as well as wildcard invalidation of assets for a domain.
• Since Photon fetches from remote origins, it does not support images hosted on private sites.

If we did find some way to support private site images and added Photon as a layer on top of the Go Files service, not only would we be caching assets twice in the current configuration, but then the cache invalidation would become the concern.

[scottsweb]

Rough plan discussed at meetup:

• Disable Photon Module
• Revert our changes that force the photon module to use the Go domain
• Implement srcset style images using the VIP Go file service
• We need to see if we need to parse the intermediate file sizes filename and convert it to a query string

[m]

@darthhexx so the Go files service is as fast and performant as Photon?

[darthhexx]

@m in short, yes it is. We have the same utilities performing the compression processing on the different image types, as well as edge caching of assets in the same POPs as Photon.

There are some differences in individual image processing, due to the that fact that we support lossy compression and different default image quality settings on Photon, but this detracts from the fact that, with regards to speed, the image delivery architecture is on par with Photon's.

As a side note, WP.com and VIP Go run on the same image processing codebase and we are in the process of integrating the additional Photon processing features into that codebase (e.g. brightness, contrast, colorize, smooth, command chaining, etc.), after which Photon will also move onto the same codebase. Then not only will VIP Go and Photon use the same delivery architecture, compression utilities, and processes, but the same unified codebase as well.

[bazza]

Why not just put the images through photon normally?

One of the other reasons that we built the Go files service instead of relying on Photon is so that we can reduce the number of required HTTPS connections and take advantage of pipelining in HTTP/2. There is some possible additional overhead because of cookies that exist on the "main" domain that would not exist on i[0-2].wp.com but with header compression in http2, the overhead should be minimal. In theory, the Go files service should be faster for Go-hosted files than Photon.