A client wanted to test a software designed to improve site speed. At the end of the trial period for the software, I provided a rundown of the results of the trial along with recommendations for next steps. I’ve anonymized anything that would identify the client or the software because this isn’t intended as a software review, but as an example of the kind of testing and reporting Team Five can do for you. I’ve called the test software Software X.
This client has a somewhat complex site structure, with much of their content on a WordPress install and products on Shopify, all under one domain that appears seamless to a site visitor. This testing was done only on the Shopify side of the site.
Executive Summary
Software X is awesome and the software probably does wonders for a particular kind of site. It isn’t especially relevant to your site for a number of reasons, and I don’t think you should pay for it. Additionally, I don’t think overall site speed is a high priority to focus on at this time, though we may want to look at optimizing the WordPress side of the site as a whole.
Details
Mobile and First Time Visitors
Software X doesn’t work on Safari, which means it doesn’t work on iPhones. In the time span since we turned on Software X (Oct 22nd), more than 50% of our traffic has accessed the site via Safari. So less than half of our traffic is being affected at all.
They do plan to roll Safari into Software X in the future, but it’s off for now without a clear date for activation. Safari (in-app) will never work with Software X.
Software X also doesn’t work on first-time visitors. The caching that Software X uses is based on using the actual browser on the visitor’s computer to speed up their experience. But if the browser hasn’t ever visited the site before, there’s no way for it to have stored the information to re-use. So that’s about 75% of our visitors.
Now the New Visitor thing is a little tricky, because if a new visitor clicks to a lot of different pages on the site they will begin to experience some acceleration. So if a visitor clicks to 10 different pages they’ll experience the site as slightly faster over the course of those clicks. This is an important detail that’s going to come up again later.
But the main takeaway here is that the large majority of our users are not experiencing any effects from Software X at all, either because they’re on mobile or because they leave the site before Software X has a chance to do anything for them.
Overall Site Speed
In context of overall site speed, we haven’t seen any improvement in overall site speed since implementing Software X. Below is the average site speed since we switched to the new theme. The arrow indicates where Software X was turned on.
Here’s a wider perspective from the last 6 months. The first arrow shows where we went live with the new theme, the second one is Software X.
So we can see that site speed has remained relatively stable, with a slight overall increase a few weeks after the first arrow.
If we look at exclusively Chrome (where 90% of our non-mobile users are), we can see that site speed has been about 56% faster overall for Software X users (blue line) than for non-Software X users (orange line), although on several days they were about the same, and yesterday Software X users actually had a slower page load time than non-users.
That effect on only Chrome users hasn’t been enough to move the overall site speed in a significant way.
We can also see that on product detail pages, Software X users are actually seeing longer load times than non-Software X users. This is for just the Product A page, and it’s showing the stats from Oct 22nd to now.
You can see that Software X users are seeing more than an extra second of load time on that page. Product B is similar (1/2 second slower for Software X users).
Revenue and Conversion Rate
We also haven’t seen any significant change in revenue over this span of time, which of course is our main goal. This chart shows revenue for the last 6 months. First arrow is the new theme, second arrow is Software X.
There is a misleading graphic we *could* use to show that Software X is performing well, and it’s this one, showing that accelerated users have an astronomically higher conversion rate than non-accelerated users.
When fans of Software X report crazy better conversion rates on accelerated users, I think this is what they are talking about. The tech from Software X explained why we need to straight up ignore this statistic. (He actually called this stat “disingenuous”.) He explained that the reason this shows such a gap is that by definition the people who are more likely to buy anyway are the same people who get accelerated by Software X. This is basically the same thing I brought up in the preliminary email, and I was really impressed that he brought it up without me even asking about it – made me think they are a truly good upstanding company. Because Software X only works on people who are returning visitors and/or are spending more time on the site, it targets people who….well, visit the site repeatedly and spend more time on the site. In other words, it targets people who were more likely to convert anyway.
Cool Resources Showing Real User Site Speed
He also showed me some resources I didn’t previously know about, which are also very cool, and I actually recorded this as a quick screencast to show you the take-aways from these tools.
(I removed the actual screencast here, but it’s a walk-through of crux.run which is a super cool tool and you should go run your own website through it and see what it tells you. In the screencast, I show that Google is reporting a consistently fast load time for the Shopify side of the site.)
Big Picture and Recommended Action
And here’s a big picture view on site load time. In 2016 your average weekly sales were about double what they are now, and your site load time was about triple.
Given all of this information, I do not think that Software X would be a good investment for you at this time. I also don’t think that prioritizing site speed optimization in some other way should be on your list of top priorities, except in the context of improving the functionality of the blog/Wordpress side of the site by converting it back to Divi from the current custom theme. Site speed is always important and something that should be on our list of long term goals and things to keep an eye on. However, given your very limited staff and the plethora of issues we have available to us to address in the short term, I do not feel that site speed should be something we spend limited resources on at this time.
Recent Comments