[PLEASE NOTE that many of the website domains discussed in this report remain compromised and likely contain malware. Do not visit them unless you absolutely know you have the appropriate safeguards in place on your own computer]

UPDATE — 03/23/18 — This report has been updated to include new information on the 8 identified Grizzly Steppe IPs found in the server logs. Six of the 8 Grizzly Steppe IPs are now confirmed as Tor exit nodes instead of three at the time of this report.

Recently in mid-December we reported on unusual Russian and Kyrgyzstan based web traffic coming to the City of Ashland, Wisconsin website server starting on March 16th, 2016. The unusual nature and circumstances were such that we reported it to local authorities who in turn reported it to the FBI.

This unusual pattern of traffic starting on March 16th has been confirmed by others in the area to have occurred in at least two other Northern Wisconsin area cities — Bayfield and Washburn. We have evidence to believe it likely also occurred in La Pointe Wisconsin as well.

What led us to investigate this traffic in the first place was all the talk at the time about Russia’s potential interference in our presidential elections. This prompted us to take a more careful look at the various sources of foreign traffic coming to the City of Ashland’s website. What we found was entirely unexpected. Traffic from both Russia and Kyrgyzstan showed sharp, unprecedented increases to the City website starting around March 16th 2016 and this traffic remained elevated and sustained through the election (figure 1). This traffic pattern did not occur on any other non-government or non-municipal websites in the area that we were able to analyze. This made the results all the more odd.

Initial analysis done on the website files and server log files could not identify any compromise. Further analysis on the server log files did however identify numerous attempts to compromise the site and/or look for the presence of malicious files previously uploaded. Initial analysis identified malicious attempts in March coming from the following countries:

Russia
Kyrgyzstan
Ukraine
Czech Republic
Romania
Latvia

Many of these attempts appeared to be automated and potentially coming from a botnet as the requests were occurring many times a second and in some cases from rapidly changing IP addresses and user agents. Because none of the attempts appeared to have been successful and absent any other evidence or data we ended our analysis of the log files and focused more on why this started in March and what aspect of the US election season this would correspond to.

This all changed a few weeks ago when the Grizzly Steppe JAR report was released jointly by the Department of Homeland Security and the FBI. This report documented some of the government’s findings on the purported involvement of the Russian government’s interference in the US elections via cyber means. Part of the Grizzly Steppe report discussed an IOC (indicator of compromise) which was a PHP based webshell used to establish sustained access and compromise of a website including WordPress based websites. It was of the type of webshell that we routinely look for on our clients websites. We quickly updated our scanner with a new regex based code fingerprint and verified that this particular webshell was not present. Very good analysis on this webshell has already been done by Mark Maunder at the company Wordfence. Many of the malicious attempts to compromise the City of Ashland website, that we found in the server logs, were of the type that if successful would subsequently lead to the upload and installation of a webshell on the website server. It should be noted that the webshell described in the JAR report was NOT the same as the malware found on the DNC network by the company CrowdStrike. There is alot of confusion in the media that the less sophisticated webshell described in the JAR report was the same malware that infected the DNC. The two pieces of malware are very different and serve very different purposes. The other part of the JAR report that provided new information was a list of over 800 IP addresses that our government said were used during the Russian hacking operations.

We cross-checked this list against the server logs from the City of Ashland and found at least 8 IP addresses from their report that exactly matched IP addresses that were probing the City of Ashland website back in March of 2016. I am not sure how to calculate the odds of this happening by random but considering there are roughly 4 billion IP addresses in use at any given time the odds are astronomical that these Grizzly Steppe IP addresses just happened to randomly also hit the City of Ashland website in March. Of these eight IP addresses six were confirmed TOR exit nodes, which is an encrypted and anonymous browsing service that is essentially untraceable. The list is shown below with the identified country of origin for the IP address.

171.25.193.235 — tor-exit3-readme.dfri.se — Sweden (tor exit node)
178.162.211.216 — germany.privateinternetaccess.com — Germany, Leaseweb GmbH (vpn)
193.90.12.86 — tor-1.multisec.no — Norway (tor exit node)
212.117.180.21 — ip-static-212-117-180-21.server.lu — Luxembourg, XBT / Webzilla
212.47.227.72 — tor-exit.hermes.bendellar.com — France (tor exit node)
212.83.40.238 — NA — Germany, 23Media GmbH (tor exit node)
85.248.227.163 — ori.enn.lu — Slovakia (tor exit node)
93.115.95.202 — lh28409.voxility.net — Romania (tor exit node)

What was interesting about these entries in the server logs were that they primarily appeared to be referral spam. Basically entries that were injected with fake referrer data and sometimes fake user agent data. Because most of them were aimed at the non-www version of the city website this request caused an immediate 301 redirect to the “www” version. In many cases this in turn generated a request from a new IP address. Based on the groupings of the requests there seemed to be a set of 10-15 different domains that were used repeatedly throughout March.

Other IP entries (non-Grizzly Steppe matched) showed a more garden variety type of request that had the goal of testing whether the website had already been compromised with a previously uploaded malicious file. This block of entries made 10 requests in 3 seconds from 7 different rotating IP addresses. Some of these IP addresses were from the same networks identified by the Grizzly Steppe report but were not exact matches.

What was also interesting to us was that the vast majority of all the entries described in this report came from 1 of 2 user agent profiles. These could be identified using the below regex pattern.

regex = Mozilla\/5\.0 \(X11; (Ubuntu|Linux)

When we searched for this pattern against the city server logs in March we found even more entries that were consistent with what we identified in the Grizzly Steppe report. That leads us to believe that the +800 IP addresses identified in the Grizzly Steppe report was by no means an exhaustive list. Below shows a couple of new IP addresses we identified.

So we already knew that one of the goals of these requests to the City of Ashland website server back in March and through the election was to look for vulnerabilities, exploit the website and potentially insert malware into the website. These newly identified requests in the server logs appeared to be solely referral spam and was an entirely different campaign. Generally, referral spam has a simple financial goal of getting additional traffic to be referred to the domains that the owner has created which could be ecommerce sites, porn site or simple parking pages with Google Adsense campaigns running on them. That is usually done by a single operator running a couple of websites. What was odd about this was there were so many different domains being generated in rapid succession and more importantly they were using IP addresses identified in the Grizzly Steppe report. This type of referral spam would also only really be viewable by a website administrator who is generally the primary person reviewing the webstats. Seeing a significant uptick in traffic coming from one of these strange referral sites might entice an administrator to click on the link and check out why this site was sending traffic to their particular website.

When we looked at who owned these domains, when they were registered and where they were hosted we noticed more Russian ties. We could identify at least two individuals responsible for these domains: Wang Wu of Chengdu, China and Svetlana Dobrynina of Tokyo, Japan. A third group of domains had their registration information private. Interestingly one of Wang Wu’s domains was later made private through a Russian privacy registrar.

Group 1:

http:// hundejo(dot)com –> Location: Chengdu, China Name: Wang Wu RegistryGate GmbH DNS: ns1.dsredirection.com (owned by Wang WU since April 2015)

http:// burger-imperia(dot)com –> Location: Chengdu, China Name: Wang Wu RegistryGate GmbH DNS: ns1.dsredirection.com (owned by Wang WU since May 2015 – then in July 2016 this was then made private using a Russia Privacy service with a Moscow address, IANA=1606. DNS remained as ns1.dsredirection.com)

http:// pizza-imperia(dot)com –> Location: Chengdu, China Name: Wang Wu RegistryGate GmbH DNS: ns1.above.com (owned by Wang WU since April 2015)

http:// hvd-store(dot)com –> Location: Chengdu, China Name: Wang Wu RegistryGate GmbH DNS: ns1.dsredirection.com (owned by Wang WU since April 2015)

http:// pizza-tycoon(dot)com –> Location: Chengdu, China Name: Wang Wu RegistryGate GmbH DNS: ns1.dsredirection.com (owned by Wang WU since April 2015)

Group 2:

http:// azartniy-bonus(dot)com –> Registered using a Hong Kong Privacy company. DNS: ns1.skydns.net (first created in May 2015)

http:// sale-japan(dot)com –> Location: Tokyo, Japan Name: Svetlana Dobrynina (Russian National with Russian email address) Company: Nikko Capital Registar: REGTIME LTD DNS: ns1.nameself.com (first created Dec 2014)

http:// bio-japan(dot)net –> Location: Tokyo, Japan Name: Svetlana Dobrynina (Russian National with Russian email address) Company: Nikko Capital Registar: REGTIME LTD DNS: ns1.nameself.com (first created Dec 2014)

http:// bio.trade-jp(dot)net –> Location: Tokyo, Japan Name: Svetlana Dobrynina (Russian National with Russian email address) Company: Nikko Capital Registar: REGTIME LTD DNS: ns1.nameself.com (first created May 2011 — last updated May 15, 2016)

http:// xrus(dot)org –> Location: Nassau, Bahamas Private Registration. DNS recently changed to ray.ns.cloudflare.com (first created in Dec, 2014 — Changed to Private Registration info on March 22, 2016)

http:// royal-betting(dot)net –> Location: Nassau, Bahamas Private Registration. DNS: arnold.ns.cloudflare.com (first created in Apr, 2015)

Note: xrus(dot)org appears to be a Russian porn site and was likely compromised at some point in the past.

Below is an example of what one of these parking domains looked like.

When we further analyzed the structure and code on these various domains we found a few to be using outdated CMSs that had well known security vulnerabilities and some that loaded suspicious javascripts and/or PHP files. Most seemed to either be compromised or had indicators that they had been compromised in the past. The hvd-store(dot)com domain in particular caused a cycling of redirects in our browser until it sent us to a URL hosted on lazymae(dot)com that was clearly intended to compromise our computer with a fake Adobe Flash update download.

It is likely that many of these other domains are or were infected with malware with the intention of infecting or compromising the user’s computer who might unwittingly download what they thought was a valid software update. It is our hope that the vast majority of administrators and IT specialists would be experienced enough to know this was fake .. but it only takes a few people to make a mistake. There have been some 0day exploits found in web browsers (like the recent Firefox exploit) so some admins could have been compromised by just clicking on the links if they weren’t using the most up to date and secure web browser.

To the best of our knowledge this is the first time we have seen the coordinated use of referral spam solely to target website administrators and attempt to compromise their computers with malware. The goal would be to intercept communications and capture administrator credentials that could then be used to compromise other networks and systems under the administrator’s control.

It is our belief that the foreign based attack on the City of Ashland website server was a two pronged attack. One goal was to target us as the website administrator and the other goal was to probe the website and server for vulnerabilities that could then be exploited.

What is still unanswered is whether the owners of these referral spam networks were aware of what was going on or were they also compromised and used as a proxy network to further the goals of the Russian government.

Eric Company News, Cybersecurity, Digital Investigations, Website Security