Google’s Safe Browsing program protects a lot of people from phishing, malware, and other harmful sites. In fact, it’s so pervasive that it’s not just used by Chrome. Google has managed to get competing browsers, Safari and Firefox, to use it as well. The list of websites blacklisted though is all maintained by Google, giving Google a disproportionate amount of power over what sites are seen versus blocked.

I’ll explain what happened during our harrowing week in a moment; but first, let me share an update about Google’s Safe Browsing API:

• Effective September 11, 2019, the Safe Browsing API is limited to non-commercial use only. It is a free service.
• If you’re a non-commercial user, you can continue to use the Safe Browsing API, and you don’t need to take any action. Also, academic researchers and NGOs will continue to be eligible to use the Safe Browsing API at no cost. 
• Commercial users, like GMass, are required to switch to the Web Risk API by November 11, 2019. This version is a paid service, and if you’re a commercial user, as we are, then you’ll need to switch. Also, you’ll need to apply and be approved to use the API, so don’t wait.
• Both APIs are directed at protecting users from malware and phishing, but the new Web Risk API is designed specifically for larger users.

And now, back to our story …

What happened?

At the beginning of last week, my team suddenly had a deluge of complaints from users reporting that their links had stopped working, their sending limits had been reduced, and they’d received phishing notifications from Google. Ouch! Upon investigating, we found that the tracking domains they use in their email campaigns had ended up on the Google’s Safe Browsing blacklist, and when people clicked links in their campaigns, they ended up on this scary bright red Google warning page.

There’s the rub. Not only does Google list actual phishing sites, but any sites that re-direct to them as well. That makes it so that any providing click-tracking services, including URL shorteners and email marketing service providers, have to be extra cautious.

In slogging through our logs we found that a Netflix phisher had managed to send 200 emails through GMass on Sunday, February 17. Here’s the email he sent. Notice that the URL behind the link is highlighted at the bottom. In this case, the URL https://www.gregsimages.com/cgi-bin led to the red screen of death, as did our click-tracked links that redirected to this URL.

The effect of being listed goes beyond just the web

What happens to your email campaigns when you send a URL that’s on this list? Gmail makes you feel like the devil reincarnate.

1. Your Gmail account becomes severely limited and unable able to send any substantial volume of emails.
2. If you do manage to send an email with a blacklisted link, and that link is clicked, you’ll scare the bejesus out of them with the red screen of death.
3. Your past campaigns are affected too, if someone pulls up one of those old emails and clicks a link.
4. If people reply to you, the reply will still have the original email at the bottom, with the link that you sent, and Google will flag the reply to you as a potential scam too.
   

Obviously not a good look for an email marketer.

What was actually affected?

In our case, when we got the notice that link.wordzen.com was listed, I wanted to isolate which link.wordzen.com URLs were affected.

Using the public Safe Browsing lookup tool, which is different from the private tool everyone has in their Search Console, I found that plugging in just link.wordzen.com gave this result, indicating that some URLs were affected and others weren’t:

But if I plugged in our original click tracking structure, link.wordzen.com/x/c, then the issue was clear that all /x/c links were blacklisted

Curious, though, I wondered what would happen if I changed the URL slightly, so I tried link.wordzen.com/x/d and found that those URLs came back clean

You’ll see below that this discovery led me to change the structure of a click-tracked link.

Some nonsense associated with the Safe Browsing Program

1. Despite claims of transparency, it’s anything but.

As soon as your domain has been listed, you will probably want to know the URL that caused the issue. Unfortunately for you, Google won’t tell you. In the Search Console you might see this:

If you received an email notification about the blacklisting, that will have a portion of the URL.

For my purposes, this did not help at all, because every click-tracked link in the history of GMass starts with /x/c. The important stuff is all after that part, and that’s the part that Google eliminates from its reports. How this qualifies as transparency is beyond me.

So what do you do now? You crawl through your logs and figure out what happened. In my case, I ran all sorts of analyses, starting with a time distribution analysis of all clicks in the last 72 hours, looking for anomalies. I also discovered my new best friend, a tool from Microsoft called Log Parser Studio, that lets me run SQL queries against my IIS log files, without importing the log files into a database. Slick! That’s how I concluded that the aforementioned Netflix phisher was the culprit.

2. The Safe Browsing API has been broken since day one.

I’m not the only one who’s frustrated over the seemingly useless Safe Browsing API.

Here’s a screenshot showing the listing of link.wordzen.com.

Here’s a screenshot showing the API result:

So with a useless API, how does one monitor whether new URLs have been listed or old URLs have been marked safe? I used Upwork to quickly a hire a team of Virtual Assistants in the Philippines to manually enter all 1,400 of our customers’ tracking domains into the web lookup tool and to do so every 12 hours until further notice.

In case you’re a C# programmer and want to try this yourself, here’s the code:

            string apikey = "AIzaSyBnIDLCc*************z4twxdx28";
            string payload = @"{
                ""client"": {
                  ""clientId"":      ""gmass"",
                  ""clientVersion"": ""1.0.0""
                },

                ""threatInfo"": {
                  ""threatTypes"":      [""MALWARE"", ""SOCIAL_ENGINEERING"", ""UNWANTED_SOFTWARE"", ""POTENTIALLY_HARMFUL_APPLICATION""],
                  ""platformTypes"":    [""WINDOWS""],
                  ""threatEntryTypes"": [""URL""],
                  ""threatEntries"": [
                    {""url"": ""http://link.wordzen.com""}
                  ]
                }
              }";

            using (var client = new HttpClient())
            {
                var content = new StringContent(payload, Encoding.Default, "application/json");
                var response = client.PostAsync($"https://safebrowsing.googleapis.com/v4/threatMatches:find?key={apikey}", content).Result;
                string resultContent = response.Content.ReadAsStringAsync().Result;
            }

There is also a .NET client library for the Safe Browsing API, but I chose not to use it because I wanted to be in control of the exact JSON payload request so that I knew that I wasn’t making a mistake.

3. Reviews take a long time to process.

Despite Google’s claims that a phishing URL will be reviewed within 24 hours and a malware URL within 72 hours, I found it took more like 120 hours (5 days). After a week of hell though, there’s no sweeter email one can receive than this, clearing your domain off the dreaded blacklist:

You can submit each URL for review, and in our case, we did it for hundreds of our customers too.

How I worked around the blacklisting of 300 domains

During the time our customers’ domains were listed, we couldn’t just sit idle waiting for Google, because customers needed to send their email campaigns out. We took a number of counter-measures and you can take these too, should you find yourself in this situation.

1. Removed all affected customer tracking domains from their accounts

The first step we took was to remove all the affected tracking domains from users’ accounts, causing them to fall back to our shared tracking domain, which for paying users, is a clean domain based on Amazon’s AWS domain. That eliminated the issue in future campaigns.

2. Altered structure of click tracking link

I felt a little shady making this change system-wide, but it turned out it worked, and saved me from an ever-growing mob of angry users. The old structure of a click-tracked link looked like:

http://link.wordzen.com/x/c?c=2724793&l=dedb940f-55dd-4c33-a3cc-f5d8028183ca&r=ed06197e-cf9e-4fea-a371-d5f455968519

By plugging various combinations of URL structures into the lookup tool, we noticed that the link above was flagged as dangerous, but if we made a simple change, changing the /c to /d, then the URL came back as safe, so we changed the structure of links to use /d instead of /c, like this:

http://link.wordzen.com/x/d?c=2724793&l=dedb940f-55dd-4c33-a3cc-f5d8028183ca&r=ed06197e-cf9e-4fea-a371-d5f455968519

Of course for backward compatibility, we made sure that the /c URLs still worked also.

Making this change allowed users to keep using their custom tracking domain, even if their tracking domain was still listed in the Safe Browsing system as a potential phishing domain.

3. Locked down how click-tracking works to prevent future issues

I should have implemented this from day one, but I could have never expected an evildoer to exploit this loophole. You may know that to set up a custom tracking domain, you create a sub-domain off your existing domain and set its CNAME to x.gmtrack.net. So in my case, link.wordzen.com has a CNAME to x.gmtrack.net and that’s what allows it to function as the tracking domain for my email campaigns. But let’s say that you set up link.yourdomain.com as your tracking domain. Before, you could substitute the domain portion of a URL with a different tracking domain, and the redirect would still work.

For example, you could substitute the “link.wordzen.com” portion of:

http://link.wordzen.com/x/d?c=2724793&l=dedb940f-55dd-4c33-a3cc-f5d8028183ca&r=ed06197e-cf9e-4fea-a371-d5f455968519

with any other user’s tracking domain, and the redirect to the GMass website would still work. This is what a recipient of the Netflix phishing email did. He substituted as many of our customers’ tracking domains as he could find into the URL that led to the phishing page, accessed those URLs, and then submitted them to Google’s Safe Browsing database.

Now, however, after locking it down, using a tracking domain that wasn’t originally associated with the link won’t automatically redirect the recipient to the website. Instead, an interstitial page will load. Try it with this link where I’ve substituted the tracking domain with a different user’s:

http://track.texthub.com/x/d?c=2724793&l=dedb940f-55dd-4c33-a3cc-f5d8028183ca&r=ed06197e-cf9e-4fea-a371-d5f455968519

TL; DR

Been blacklisted by Google’s Safe Browsing program? Here are the things to know.

1. Don’t use the API to monitor listings. It doesn’t work.
2. The information in the Search Console is mostly useless. The better information is in the email notification about the listing.
3. After fixing the issue, submit your site for review, but don’t expect an immediate resolution. It can take a week for delisting to occur.
4. While you’re waiting on #3, try to work around the issue, like we did by altering the structure of click-tracking links. Listings in the Safe Browsing database seem to be based on a regex match, not every single URL at your domain.
5. If all else fails, use this Wired article to track down the team of Google engineers that built the Safe Browsing product. There’s a list of names at the bottom. Use LinkedIn, Hunter.io, or any means necessary to track someone down.

More exciting tales…

This isn’t my only encounter with the Google Safe Browsing program. In September 2020, I faced another harrowing issue when the program blacklisted the domain we use with Zendesk. I also have an interesting story involving a Spamhaus blacklisting.