Track pageviews from browsers with blocked JavaScript or Google Analytics using Measurement Protocol

Few weeks ago I configured this site to use CloudFlare and I was astonished with the difference in analytics between Google Analytics and CloudFlare. Number of unique visitors was 4 times bigger in CloudFlare than in GA. I knew that this number could not be true, but it made me think about pageviews I am missing due to visitors with JavaScript disabled, or browser extensions which are blocking GA universal code.

Solution for this problem was straightforward, for browsers without JS I'll use noscript tag, and for the ones that are blocking GA I'll check for existence of ga object (Google Analytics object) in window object. I wanted to log all these visits through my backend, so site does not need to directly query Google servers from frontend and possibly be blocked by 3rd party extensions.

Next problem was how should I send these pageviews to Google Analytics so I can see them in my GA dashboard of my existing account. I did a quick online search and discovered great feature of Google Analytics, Measurement Protocol. In short this protocol enables you to send pageviews, events etc. directly from your code to GA, in my case from Laravel backend.

Main Layout modifications

As I mentioned above, for browsers with disabled JavaScript I am going to use noscript tag. So just under opening body tag I will insert this code:

<noscript><div style="display:none;visibility:hidden"><img src="/logme.png?category=NoJS" /></div></noscript>

I added this category parameter to url so I can use same method for browsers which are blocking GA also.

This noscript element is pretty easy to understand, if JS is disabled on that place browser will load hidden div with image. Image is not only an image, it will log that certain visit after it return 1 pixel image.

Now I will add code just above closing body tag, for logging visits for browsers with JavaScript but which are blocking Google Analytics script.

If GA is properly loaded into browser it will add ga object to global/window object. So my job here is to check is ga object present after page loads:

<script>

    window.addEventListener('load', function () {
        if (!window.ga || !ga.create) {
            var img = document.createElement('img');
            img.setAttribute('style','display:none;');
            img.src = '/logme.png';
            document.body.appendChild(img);
        }
    });
</script>

From above JS code you can see that I added listener for load event, in the callback function I am checking is window object loaded. If it is not loaded or if it's not fully loaded (ga.create function is missing) I am creating new img element, setting it to non-visible and adding same src as for image in noscript tag.

At the end I am appending that new element to body of the page.

Fake Image Route

Laravel allows me to use almost any combination for route url, so I'll make it to look like png image:

Route::get('logme.png', ['as' => 'tracking', 'uses' => 'TrackingController@getPixel']);

Tracking Controller - Backend code

This tracking controller will have only one method, getPixel(). First it will echo transparent 1 pixel image to the user, after that it'll push the data to measurement protocol servers.

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class TrackingController extends Controller
{
    public function getPixel(Request $request)
    {
        header('Content-Type: image/png');
        echo base64_decode('R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs');

        $tid = 'UA-XXXXXXXX-X';
        $ec = 'Blocking GA';
        $ea = $request->server('HTTP_REFERER');

        if ($request->get('category') == 'NoJS') {
            $ec = 'NoJavaScript';
        }

        $ip = $request->ip();
        //CloudFlare IP fix
        if ($request->hasHeader('CF-Connecting-IP')) {
            $ip = $request->header('CF-Connecting-IP');
        }

        $params = array(
            'v' => 1,
            'tid' => $tid,
            'cid' => rand(1000000,9999999),
            't' => 'event',
            'ec' => $ec,
            'ea' => $ea,
            'ni' => '0',
            'z' => rand(1000000,9999999),
            'ds' => 'web',
            'uip' => $ip,
            'dl' => $request->url()
        );

        $url = 'https://www.google-analytics.com/collect';
        $content = utf8_encode(http_build_query($params));
        $user_agent = '';

        $ch = curl_init();
        curl_setopt($ch,CURLOPT_USERAGENT, $user_agent);
        curl_setopt($ch,CURLOPT_URL, $url);
        curl_setopt($ch,CURLOPT_HTTPHEADER,array('Content-type: application/x-www-form-urlencoded'));
        curl_setopt($ch,CURLOPT_HTTP_VERSION,CURL_HTTP_VERSION_1_1);
        curl_setopt($ch,CURLOPT_POST, TRUE);
        curl_setopt($ch,CURLOPT_POSTFIELDS, $content);
        curl_exec($ch);
        curl_close($ch);
    }
}

Instead of UA-XXXXXXXX-X you should use your own tracking ID. 

In first if statement I am checking is user browser blocking JS or GA.

In second if statement I am checking CloudFlare header, and if it exists I am taking oriiginal user IP. CloudFlare will serve my site from their own servers, I'll start to see locations of their data centers in GA instead of visitors locations. Because of this CloudFlare sends additional headers in each request to original server. From these haeders I can extract CF-Connecting-IP header, and pass it to Measurement Protocol.

If you are not using CloudFlare, you can ommit this if statement.

After I compile params array I will encode the data and pass it to GA using curl.

This method actually sends events to GA, because I wanted to see how many users are blocking GA or have disabled JS. After few weeks I modified it to log pageviews, now it looks like this:

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class TrackingController extends Controller
{
    public function getPixel(Request $request)
    {
        header('Content-Type: image/png');
        echo base64_decode('R0lGODlhAQABAIAAAAUEBAAAACwAAAAAAQABAAACAkQBADs');

        $tid = 'UA-XXXXXXXX-X';
        $page = $request->server('HTTP_REFERER');
        $urls = explode('/', $page);
        $page = $urls[sizeof($urls) - 1];

        $ip = $request->ip();
        if ($request->hasHeader('CF-Connecting-IP')) {
            $ip = $request->header('CF-Connecting-IP');
        }

        //Unique visitor code
        if ($request->session()->has('ip_' . $ip)) {
            $cid = $request->session()->get('ip_' . $ip);
        } else {
            $cid = rand(1000000,9999999);
            $request->session()->put('ip_' . $ip, $cid);
        }

        $params = array(
            'v' => 1,
            'tid' => $tid,
            'cid' => $cid,
            't' => 'pageview',
            'dh' => 'tuts.codingo.me',
            'uip' => $ip,
            'dp' => $page
        );

        $url = 'https://www.google-analytics.com/collect';
        $content = utf8_encode(http_build_query($params));
        $user_agent = '';

        $ch = curl_init();
        curl_setopt($ch,CURLOPT_USERAGENT, $user_agent);
        curl_setopt($ch,CURLOPT_URL, $url);
        curl_setopt($ch,CURLOPT_HTTPHEADER,array('Content-type: application/x-www-form-urlencoded'));
        curl_setopt($ch,CURLOPT_HTTP_VERSION,CURL_HTTP_VERSION_1_1);
        curl_setopt($ch,CURLOPT_POST, TRUE);
        curl_setopt($ch,CURLOPT_POSTFIELDS, $content);
        curl_exec($ch);
        curl_close($ch);
    }
}

Above code has few modifications according to Page tracking Guide. I added small snippet for checking is user unique or returning, by storing their ip in session with random integer. This way GA will not show every pageview from same visitor like new unique visitor.

That's all, if you are too lazy to implement code by yourself you can use this package Laravel-GAMP Package. For such a simple features I don't like to pull additional dependencies and add overhead to my projects.

Exclude hits from bots and spiders

Now when we've enabled to track non-js browsers it's very likely that we'll log more pageviews from bots and spiders. To be sure that GA is only showing traffic from legitimate users you'll need to go to Admin -> View -> View Settings of your property, and look for checkbox which says Exclude all hits from known bots and spiders. Check it and save settings.

View Settings Tab

View Settings Tab

If you first want to compare all traffic with bots excluded traffic, then you can make new property based on existing one. Just click Copy view button on View Settings page, and exclude traffic from bots on that new view.

It's important to mention that views can not filter historic data, they only work on new data.

Final results 

After couple of weeks with this new setup, I discovered that ~15% of visitors which visit Tuts, surf the web with disabled JavaScript or with extensions which are blocking Google Analytics Universal code from reporting. Suddenly I got many more pageviews, just by implementing this :)

Possible improvement

If you are concerned about time delay between your server and GA servers, you could use Laravel Queues and first insert new requests into queue and after that use workers to push data to GA.

 

Online computer science courses to jumpstart your future.
WP Engine Managed WordPress Hosting

Trending

Newsletter

Subscribe to our newsletter for good news, sent out every month.

Tags