Last week, our Zendesk subdomain, gmass.zendesk.com, was flagged by the Google Safe Browsing program. This had the unfortunate effect of sending all Zendesk ticket notification emails to our customers and admins to the dreaded Gmail Spam folder. Ugh. What a mess. In case this happens to you, this is how you can fix it.

What is a Zendesk subdomain?

When you sign up for Zendesk as your support system, you’re assigned a subdomain that’s usually in the format of [yourcompany].zendesk.com. In our case, it’s gmass.zendesk.com. The subdomain makes appearances in all email notifications to/from requesters and agents, and is also the location to your online help portal if you have one hosted by Zendesk.

In our case, we provide support only via the email channel. But, if a domain is flagged by the Google Safe Browsing program, then it is also flagged by Gmail.

Our research has shown that emails containing domains listed by the Google Safe Browsing program go straight to Spam in Gmail.

Why did it get blacklisted?

In the case of gmass.zendesk.com, Google Safe Browsing did offer us reasons for the listing via the Google Search Console. Let’s examine those.

Google points out two “harmful download” links and one “deceptive page.” The harmful downloads are legitimately harmful — they are attachments that were uploaded as part of a Zendesk ticket by a malicious “requester.” The “deceptive page” was not actually deceptive at all. It was a screenshot of the Google sign-in screen where a legitimate user showed that he was having trouble logging into his Google account. So that was just an error on Google’s part.

More importantly, how did Google’s index grab these URLs in the first place? We don’t make them public anywhere. My working theory is that either:

1. The malicious user that created the ticket then posted the URLs to the attachments somewhere public, where they were indexed.
2. Google uses URLs in Gmail as a source of indexing.

Naturally, the solution that comes to mind is to disable those URLs. Make them return a 404. Remove the offending attachments. That, however, is easier said than done, because Zendesk does not make it easy.

Did Zendesk Support help me? Of course not.

There’s an obvious irony baked inside that question. Zendesk is a support platform, and therefore one might expect them to provide stellar support themselves. But that’s not always the case. GMass is an email automation and outreach platform, and our own outreach isn’t much to speak of (our technology though is the world’s best). Occasionally a dentist has a bad tooth.

When I first reported the issue, Zendesk pointed me to their generic deliverability article. When that obviously didn’t work, they correctly guided me to the article on adding the sub-domain to Google Search Console, and it was then that I realized the problem. Zendesk didn’t provide any help beyond that, despite their claims that they monitor the Safe Browsing list for their customer’s domains. They pointed me to articles on removing attachments (which did nothing for me) and gave me incorrect information on why sub-domains still appear in emails even after setting a host mapping.

I was on my own, but that’s okay, because I’m a pretty smart guy.

The top 3 most incorrect and unhelpful things Zendesk Support agents told me:

3. “So I just heard back from the team and was able to confirm that as far as the email behavior is concerned, one of our experts confirmed that the Gmail G-To Actions were hiding a link to the ticket in the body of the email which would be out of our control. This may need to be addressed by Google in order to let these kinds of emails pass through their server without being tagged as spam.”

Gmail’s G-To actions had nothing to do with the issue at hand, and there is no link in the body of the email, which is out of Zendesk’s control. The agent is speaking as if I can just call up Google and have them “address” the issue.

2. “The emails with the email domain gmass.zendesk.com as the sender is going to your Suspended folder, and you want to prevent this — right?”

After clearly explaining that the issue was that emails are going to the Gmail Spam folder, the agent confuses this with Zendesk’s own “Suspended” folder.

1. “I got a response from our internal team. As per them, using an external email with SPF/DKIM should remove any issue regarding that otherwise, following the information in this article will help in your case: Users see red warning screens when trying to access Zendesk or ticket attachments.”

SPF was already set up, and DKIM is completely non-applicable to the situation at hand.

I’ve tweeted before about the inadequacy of most large SaaS providers’ support teams:

I’ve noticed a drastic deterioration in the support provided by major SaaS companies like @SendGrid and @Zendesk in the last few months.

It’s a shit show, much like the presidential debates tonight.

— Ajay Goel (@PartTimeSnob) September 30, 2020

Of course, there are exceptions. I’ve received excellent support from Dropbox and LiveChat in the past. The biggest determinant of whether you get good or bad support from a SaaS platform is based on the randomness of the support agent to which your issue is assigned. How well does your support agent know the SaaS platform? The 80/20 rule likely applies to support agents: 20% of the support agents are responsible for 80% of successful issue resolutions.

Can’t you just remove the subdomain from email notifications?

That would be nice.

Zendesk provides an option to mask the sub-domain and brand the entire email support channel around your own domain. For example, I can set up support.gmass.co as a CNAME alias of gmass.zendesk.com if I want full control of branding. Except there are a few flaws in this feature.

1. Even though you brand Zendesk support around your own domain, it only affects people that send NEW ticket requests to the branded email address. If a user sends an email to the OLD address, or an agent responds to an older ticket before the change, the notification generated to the user will still contain the old zendesk.com domain.
2. Even on new tickets, while only the branded domain is visible, Zendesk, for some reason that makes no sense, still includes the domain buried in the HTML of the email notification, which is enough to trigger the Gmail Spam filter. Here’s that HTML code:

 

<span style=3D"color:#FFFFFF" aria-hidden=3D"true">[M7OQED-Z369]</span><div itemscope="" itemtype="http://schema.org/EmailMessage" style="display:none"> <div itemprop="action" itemscope="" itemtype="http://schema.org/ViewAction"> 
<link itemprop="url" href="https://gmass.zendesk.com/hc/requests/38982" /> <meta itemprop="name" content="View ticket"> </div></div>

I asked a support rep why the subdomain appears in the HTML code of the email when the whole point of the branded host feature is to eliminate it. I was told that “setting up SPF and DKIM” will fix it. Now, I never tried this, but I just don’t believe it can be true. Why? Because SPF and DKIM have nothing to do with a domain that’s buried within the HTML of the message body.

So how do you solve this?

You have two choices:

1. Change your Zendesk subdomain altogether or
2. Get your domain off the blacklist.

Changing your Zendesk subdomain is a complicated process that will be messy and cause problems. So I could have changed mine from gmass.zendesk.com to say, gmss.zendesk.com (yes, I left out the “a”). For new ticket requesters who now email [email protected], all would have been good, and the emails would have stayed out of Spam folders. However, the following would also happen:

• Anyone who responds to an old ticket would still end up emailing the old sub-domain, and that would then bounce. Zendesk doesn’t allow both old and new subdomains to work simultaneously.
• You have to change your support address throughout your website and literature from company-name@old-zendesk-domain to company-name@new-zendesk-domain.
• There’s still a chance that Google will locate the offensive content on the new sub-domain and end up listing that as well.

After asking Google over and over again to reconsider the Safe Browsing listing, and getting repeatedly rejected without reason, I finally discovered that the “sample” URLs Google provided weren’t the only URLs that Google didn’t like. I finally found the setting that let me turn anonymous access to Zendesk attachments off. It’s here, and documented by Zendesk here.

It’s frustrating, though, that neither the three Zendesk support reps I talked to nor the person that handles their Twitter account could point this out to me as the culprit. There’s a larger issue here that I’ve been vocal about recently, that well known: billion-dollar SaaS companies suck at support. Not the kind of support where the customer is asking, “How do I do this?” but the kind of support where the situation is, “Your system is failing me — how do I solve this huge problem now that I’ve come to rely fully on your software?”

So finally, after doing this, and after re-requesting a review, 24 hours later, I received this:

and this:

If you have an astute eye, you might notice the irony of Google’s email telling me that gmass.zendesk.com has been cleared while also telling me that the message is dangerous, probably for containing gmass.zendesk.com. But I’ll just chock that up to their systems not being in sync with each other.

Removing attachments from your Zendesk tickets

Zendesk provides a couple of ways to remove attachments from Zendesk tickets, but only if you know what ticket they belong to in the first place. Note the structure of the offending URLs. There’s a randomized token value, and you can see the URL encoded version of the attachment name. Zendesk allows you to search for tickets that have an attachment and tickets that have a particular attachment.

However, the list of URLs that Google shows you are only SAMPLE URLs. They aren’t a full list of offending URLs. So, if I were to remove just these three URLs and request a review, my request would be ignored. In fact, that’s what happened. I requested a review, didn’t hear back, requested a review again, didn’t hear back, did it again, didn’t hear back, and finally, I decided to cull my entire gmass.zendesk.com site for potential issues.

I searched for ALL files with an .arj extension since no user should ever send us .arj files. I also searched for all .doc files and deleted those tickets. Deleting those tickets also had the effect of disabling the corresponding attachment URLs.

Really, though, none of that was necessary.

The Actual Solution

The only thing that was necessary to resolve this, and the one thing that Zendesk support just couldn’t or wouldn’t tell me, was that I simply had to turn off anonymous attachment viewing here.

That way, when the Google crawler goes to a URL, the attachment won’t be retrieved. Instead, the crawler will be redirected to the Zendesk login page, and that’s good enough for Google to consider the problem resolved. If Google can’t crawl it, it doesn’t exist.

Zendesk resources if this happens to you

If the problem is with Google Safe Browsing, then you have to add your Zendesk subdomain to Google Search Console to determine any issues.

If the issue is like mine, where offensive attachments are the cause of blacklisting, this is how to remove attachments from tickets. Of course, you may need to use more aggressive methods, like locking down attachments altogether or deleting all tickets containing an attachment of a particular type.

Here, Zendesk discussed how they monitor issues like domain blacklistings for you and address the matter for you, but clearly, this isn’t true, at least it wasn’t in my case.

If you’ve tried everything and you can’t clear your name (or domain, as the case is here), the nuclear option is to change your Zendesk subdomain, or just leave Zendesk altogether. That might be “extra” nuclear, though. This is how you change your Zendesk subdomain. Pay attention because there are consequences.

One more thing…

Zendesk makes it incredibly difficult to NOTIFY your ticket requesters of this issue. Ideally, I want to notify the requester of any ticket that’s been updated since the blacklisting started. I can search in Zendesk:

updated<2020-09-25

But then, I can’t export those tickets to get the email addresses of those requesters. Ugh, ugh, ugh. First, only users on the Support Professional or Enterprise plans can export at all. Secondly, even if you can export, you’re limited to 100 tickets.

What I was able to do, however, was use GMass’s own build-email-list feature to search the Gmail Label where I store Zendesk notifications. From that, I generated a list of all ticket requesters during a specific date range.

Want more in-depth tales of blacklisting incidents?

This isn’t my first encounter with the Google Safe Browsing program. Also, read about my experience with Spamhaus. It’s a doozy.