Google Analytics Data Issue: Internal Site Search Terms Missing

Posted: January 15, 2019 at 1:25 pm, Last Updated: January 23, 2019 at 4:02 pm

What is Internal Site Search

Internal site search refers to searches that are done by users on your website.

Background on Core Website Internal Site Search Integration with Google Analytics

While you can integrate internal site search data with your website’s Google Analytics data, the way Google Analytics handles internal site search queries by default is limited to searches in which the search terms is indicated by a URL querystring parameter. This method does not work for all websites.

Mason Drupal Platform

When the Mason Drupal Platform was launched, we initially used data from the university’s Google Search Appliance to provide internal site search results.
However, the non-standard way the resulting site search results URLs were formed meant that we were not able to integrate the site search data into Google Analytics using the default method.
But we were able to develop a method of integrating our internal site search with Google Analytics by using a couple of custom filters to parse the search term data from the URL and add it to the ‘Search Term’ field in Google Analytics.
Since we were using custom filters for site search, we never ‘turned-on’ site search tracking on the core website. We didn’t have to. Since the default method of site search tracking wouldn’t work for us, and since we were handling site search manually using custom filters, we found that even without turning on Google standard way of implementing site search, we were able to get the search term data in the correct fields and get the site search reports running.

Swiftype

Subsequently, we implemented Swiftype to handle site search on the Mason Drupal platform.
Unfortunately Swiftype also uses a non-standard method of implementing site search results URLs, again in a way that wouldn’t work for the standard method of implementing site search tracking in Google Analytics.
Of course we were able to come up with a new custom method of integrating our site search data with Google Analytics, which in this case did require ‘turning-on’ site search tracking in Google Analytics.

Swiftype search was implemented on the Mason Drupal Platform on December 14.
The custom Google Analytics search integration was implemented on December 20th.
So we did have a gap in the internal site search data collection of about a week.

You can see this gap in site search tracking on the Google Analytics Site Search Overview report:

Note that you can clearly see that site search data was being collected before the switch to Swiftype, and the gap in tracking, and that search data was again being collected once we implemented the new integration between Swiftype and Google Analytics.

Then we left to go on winter break…

The Issue

When we returned, I opened-up Google Analytics to see what search terms people were using on out internal site search, as I typically do every week as they are a regular weekly reporting item.

I was surprised to find that our search term data from prior to December 20th was missing:

Trust me when I tell you that we used to be able to see this data in Google Analytics. As you can see by the previous search reports posts on this site, we used to review and report on this data weekly.

(The Search Pages report is similarly missing data.)

Compare these two report screenshots. They both show the same date range.
In the first, the Search Overview, you can clearly see that search data was being collected prior to December 20.
In the second, Google Analytics is reporting that is has no data on search terms prior to December 20.

So obviously these two reports are showing inconsistent data. It is not possible that searches have taken place with no search terms.

In any event, I was stunned to find that our historical search term data was missing.

Everything I have read and understood about Google Analytics indicates that there is no way to remove old data: once the data is collected it is in your Google Analytics view for all time – unalterable. There is simply no way for a user to remove or modify old data.

To add to the inconsistent behavior, if you include a custom segments (in this example I have added custom segments showing on- and off-campus traffic) the search term data from prior to December 20 reappears!:

To me this indicates that this is a problem on Google’s side.

Update: 2019-01-16
Building on my observation that the missing search terms re-appear when a segment is applied, I have discovered that if I create a segment that includes all traffic, I appear to be able to see all of the search term data.

To accomplish this, I built a custom segment that includes both on- and off-campus traffic:

Note that this segment includes all traffic.

When applying this segment, the missing data (all of it, unsegmented – because its a single segment that includes all traffic) becomes visible again!:

Communication with Google Analytics Support

I reached out to Google support to see what they could tell me:

Here is a record of my correspondence with Google Analytics support on this issue:

The fact that they have been looking into it this long, along with the inconsistencies in the Google Analytics reporting, indicates to me that this is a real issue.