Bluecoat check fails

[fastlorenzo]

Bluecoat recently modified their website, and as a consequence the method to fetch the data changed. They added an XSRF token.

There might be a way to modify the script to make it work with their new model, I'll have a look at it. In the meanwhile I would recommend removing the script part that checks Bluecoat.

[chrismaddalena]

Thanks for the update on this. We will take a look at this for Shepherd and Ghostwriter.

[linuxkd]

There is a fix for this that I have implemented in another tool. I will likely put together a fix for this and issue a PR.

[kkirsche]

@linuxkd is that fix publicly available for others to leverage?

If done via site review the URL pattern is:

http://sitereview.bluecoat.com/#/lookup-result/DOMAIN_TO_LOOKUP

From here the category will be in the span with class “clickable-category” with that said it’s a javascript based react app so you’d need something like spynner or another javascript processor to resolve it into that html

[kkirsche]

Can also hit:

Request URL: http://sitereview.symantec.com/resource/lookup

with a payload: {"url":"duckduckgo.com","captcha":"","key":"NEED TO SEE IF THIS IS STATIC"}

and get the JSON response as well

[rundro]

any update on this? looks like bluecoat, talos, Google SafeBrowsing, and Phishtank are also failing

[chrismaddalena]

@rundro The Shepherd project has been retired in favor of Ghostwriter. Ghostwriter includes all of Shepherd's features and functionality and new features and enhancements from the past two years.

https://github.com/GhostManager/Ghostwriter

That said, this issue would also affect Ghostwriter, but we removed it a while back. The pages used for these checks have changed in various ways over the years. Bluecoat has been very aggressive with their changes. The last time I checked, the anti-automation checks had started looking for two separate Base64 blobs that decoded to variations of "if you can see this, knock it off." Symantec even started banning IP addresses suspected of automated activity. Other vendors changed to requiring accounts or limiting five requests per day by IP address.

We always felt what Shepherd did was respectful and non-malicious. We included healthy delays between requests and tried to avoid causing unnecessary spikes in traffic. Even so, it became evident that the vendors really wanted us to stop. Maybe not us specifically, but we were in an arms race either way. We entertained the idea of enabling cloud services to proxy traffic (e.g., FireProx), but that would be a lot of work that would only continue the cat and mouse game.

Out of respect, we ceased using vendor webpages to collect this data and switched to using VirusTotal's v3 API to collect categorization and other data. This expanded categorization data beyond a built-in set of vendors and is 100% reliable. It may not always include real-time categorization data from a vendor like Symantec, but that can be collected manually or via another tool and updated in Shepherd or Ghostwriter.