Decreasing revenue with AdSense

A few months down the road, there was one major problem. I was still slowly loosing revenue after the implementation. Occasionally in huge percentage chunks. I was busy with 3. Semester of my education, and did not have time to debug.

I did notice at some point, ads not showing on a few pages. But, I was able to get them to show after hitting refresh a few times – so I did not think this was the problem (at the time).

Sometimes, revenue would also increase compared to the last month, which further added to the complexity in debugging the issue. I thought that I simply had a bad month.

Finally, this January, I lost as much as 70%. I realized, clearly something must be wrong with my consent mechanism.

It was then I noticed, testing in a Chrome incognito Window, ads not showing for most new articles. No matter how much I hid refresh.

InAppBrowsing is the enemy

Another problem I noticed has to do with InAppBrowsing (I think it's called). When you share content on social media, such as Facebook and Twitter, and a user visits a link you posted, the link will be opened inside a "in-app" browser window.

On Facebook Mobile Browser, there is a bug that prevents the browser from accepting cookies properly, which means that it does not remember consent. Possibly, the cause for this is that I am sending a AJAX request from JavaScript to a PHP script (on same domain name), which then sets the cookie. I am not 100% sure this is the problem, but it is my best guess. I will have to test what happens if I set the cookie from JavaScript instead.

Long story short. Facebook and Twitter simply need to stop using InAppBrowsing if they can not get it right.

I have not yet tested Twitters in-app browser, to see if cookies are working correctly. BUT, point is, there are all sorts of UX problems with these browsers as well, so there are really no reason for them to exist.

As a user, something that has frustrated me a lot, is that you can not change the URL (or even see the full URL). So.. Please stop using InAppBrowsing if you work for social media and you read this.

Why ads stopped showing

I am guessing the AdSense crawler does not always identify itself when visiting pages, and this caused ad-code not to be properly inserted into pages (depending on user-agent). Some older pages was still displaying ads, since they had already been crawled.

Google would also have to crawl my pages from different locations, including from within the EU, since the consent dialog is only shown to EU users (based on Maxminds GEO-location database).

There has been some talk about this online in webmaster circles. Displaying different content to crawlers from what you show to users, is not allowed according to the AdSense terms. So we also have to be careful what we do.

To detect something like this, obviously Google needs to also visit your page anonymously at times.

I am not sure what I did was allowed, hence why I also asked John about it on Twitter back in June:

@JohnMu #GDPR may be dangerous for #SEO - some websites might implement "consent walls" to block content, as this seem to be the safest way towards compliance. Showing a wall to users, and something else to Google (content) is against Guidelines. Is a wall safe for GDPR purpose?

— Jacob Kristensen (@JacobSeated) June 6, 2018

But, it was the best solution I could think of at the time, and I can still see some uses for hiding ad-code for specific pages. For example, on pages such as About, Contact and Privacy Policy.

Conclusion

Removing ad-code entirely from the HTML seems like a bad idea in most cases, as it might prevent Google's crawlers from finding the code.

I did originally anticipate that this could happen, but I thought user-agent sniffing could solve the problem. However, it seems that did not work very well for me.

Instead, using JavaScript, you can now use pauseAdRequests=1 to temporarily pause loading of ads, and then later call pauseAdRequests=0 when the user clicks a consent button. The implementation is actually very easy.

Somewhere in your head, place a script with this code:

(adsbygoogle = window.adsbygoogle || []).pauseAdRequests = 1; // Pause loading of ads

Then, call a loadAds() function when a consent button is clicked:

function loadAds() {
  (adsbygoogle = window.adsbygoogle || []).pauseAdRequests = 0; // Resumes loading ads (un-pause)
}

This will still need to be combined with a server-sided script to record the consent, and to place a cookie on the users device, which will remember their choice. You can even make it show non-personalized ads, though I see little reason to do this personally.

Not including ad code in the HTML can still be useful for specific pages, such as about and Privacy Policy.

Links

1. Ads personalization settings in Google’s publisher ad tags - google.com

Tell us what you think: