browsertime: GeckoView vs system WebView on the Fire TV Stick 4k, Pendant, and Cube

browsertime reports that GeckoView is slower than WebView

When running with Turbo Mode disabled, GeckoView is perhaps 10% slower than WebView on the more powerful FireTV Stick 4k and Fire TV Pendant and perhaps 50% slower than WebView on the underpowered FireTV Cube. It seems that GeckoView is also much more variable than WebView – perhaps as much as 200% more.

The performance gap does decrease with Turbo Mode enabled: GeckoView is perhaps level with WebView on the FireTV Stick 4k and Fire TV Pendant and perhaps 40% slower than WebView on the FireTV Cube. However, GeckoView becomes significantly more variable than WebView. This is very surprising because the set of processed resources should be smaller than with Turbo Mode disabled, which one expects to translate to less variability.

browsertime reports that Turbo Mode improves performance at the expense of variability

With GeckoView: enabling turbo mode produces a performance increase, more significant on the underpowered FireTV Cube, and a significant noise increase.

We see the same general behaviour with WebView: a performance increase on the underpowered FireTV Cube and a noise increase.

Turbo Mode enabled

Turbo Mode disabled

Vehicles tested

The data were collected from the following vehicle configurations:

All vehicle configurations shared a single User Agent string. See this issue.

Sites tested

22 of the 25 sites in the product mobile corpus were tested. The sites not tested were:

Some sites witnessed transient network errors: in these cases the number of recorded measurements is fewer than expected. In any individual run, no site was measured fewer than 4 times on the Fire TV Pendant and Cube, or fewer than 8 times on the Fire TV Stick 4k.

The entire corpus was tested end-to-end twice in succession on the Fire TV Pendant and Cube, and end-to-end once on the Fire TV Stick.

Single site test

For each site, the four vehicle configurations were tested as follows:

1. An initial recording of the live site was captured. The record and replay proxy was started in recording mode, and browsertime with --iterations 1 launched the vehicle and (cold-)loaded the site under test. The replay proxy was stopped and an archive of the network activity captured.

2. The record and replay proxy was started in replay mode, backed by the archive of captured network activity. browsertime, with --iterations 5 (respectively, --iterations 9) on the Fire TV Pendant and Cube (respectively, Fire TV Stick 4k) launched the vehicle and cold-loaded the site under test the specified number of times. Between each cold-load the vehicle was force-stopped and its on-device package-data cleared.

3. For each cold-load, browsertime reports a wide range of timings, mostly from the Performance Navigation Timing API.

Raw data

The data collected for the following tests is available:

Processed data

The data from the three test runs above can be found in the following CSV file. The columns of the CSV are as follows:

Test harness

The data were collected using an ad-hoc Python harness driving the browsertime testing suite. browsertime drives the underlying vehicles using Web Driver automation; for WebView this means chromedriver driving the engine via the Chrome Debug Protocol and for GeckoView this means geckodriver driving the engine over the Marionette protocol.

The version of browsertime used was lightly modified to support Android-specific WebView engine configuration and to support the GeckoView engine. None of these modifications are believed to impact engine performance.

The version of geckodriver was heavily modified to support the GeckoView engine over the adb TCP/IP protocol. These modifications principally concern launching the target vehicle and connecting to the underlying protocol handler; any impact on engine performance has to do with servicing the underlying protocol and ambient engine configuration (for example, custom profiles in GeckoView).

Network weather

Both mitmproxy and Web Page Replay Go were used to minimize the impact of network weather. Because older versions of adb do not allow to reverse port-forward over TCP/IP [link], the test host and the target device were always on the same network. Because Web Page Replay Go is not a true HTTP proxy [link] but instead requires transparent port-mapping [link] and because Gecko does not support such port-mapping [link], mitmproxy was used to perform the port-mapping [link to script]. Record and replay were provided by wpr-go, although it is likely that mitmproxy could provide this function.

Using a proxy and a custom CA certificate for both WebView and GeckoView sacrifices real-world characteristics for cross-engine consistency. GeckoView requires a true HTTP proxy for this type of record and replay, and such a proxy requires either a custom CA certificate or for the engine to allow insecure connections. Allowing insecure connections is decidely not real-world, hence we took the lesser of two evils.

Record and replay differences

The turbo mode option should change the network activity captured by the record and replay proxy. However, it is also possible that the two engines witness different network activity – for example, by User Agent sniffing sites. This means that each individual site and vehicle configuration should have stable network activity, but between vehicle configuraitons there could be network activity differences.

Live site differences

Some of the sites serve dynamic content and/or advertisements. This means that between the first and second whole-corpus iteration, the underlying network archives may have changed significantly.

Gecko profile conditioning

It is well known that the Gecko profile significantly impacts the performance of the Gecko engine: preferences, certificate databases, and the network cache itself can have major impacts on measurements.

To minimize volatility, for each GeckoView-based vehicle configuration, i.e., for both turbo enabled and turbo disabled, a Gecko profile was conditioned as follows. First, a profile template with cert9.db and key4.db containing the custom CA certificate used by the record and replay proxy was produced. Second, this template was copied to the target device, and the vehicle was started from a cleared state with this profile. The single page http://example.com was visited and then the vehicle was left idle for 2 minutes. The vehicle was then force-stopped and the conditioned profile retrieved from the device.

This conditioned profile was then copied to the device at the beginning of every test run: that is, every cold pageload started with exactly the same Gecko profile.

Fire TV versions

The output of adb shell getprops for each device is available:

Software versions