Threat Crowd

Thursday, 26 January 2017

New job at AlienVault OTX

It turns out I enjoy working on a threat intelligence platform so much I’ve decided to do it full time - and have moved to AlienVault to work on their OTX platform. You can read more about that over on the AlienVault blog. OTX already has some very strong social and integrations features, and I’ll be particularly working to add in more of the analysis tools that people have found useful in ThreatCrowd.

The obvious question I anticipate is “What will happen to ThreatCrowd?”. The overall plan here is that the community will have free access to more tools and more data - not less. I’m looking forwards to working on a platform that doesn’t have ThreatCrowd’s constraint of constantly running at 99% diskspace on an 80 GB Linux web server!

There are things that aren’t available in OTX yet such as the graph, whois/ssl/nmap data and some of the API functionality. Much of that is already in development on OTX, and the rest should be within reach.

Development on ThreatCrowd ceased about a year ago when I reached the core aims of the protoype I had in mind, and I haven’t yet worked out an exact roadmap for ThreatCrowd.org yet. I imagine I’ll migrate parts over as new functionality gets introduced to OTX. Again the broad principal here is that the community will have more access to tools and data, not less.

I’ve had a ton of feature requests for ThreatCrowd, many of which I haven’t had time to reply to (sorry!) let alone implement. But if you’ve you’ve got any ideas now’s the time : )

You can suggest features in OTX by clicking “Feedback” here

Thursday, 24 November 2016

Building a Whitelist of Network Domains

There are a number of times when a white list is useful to security professionals, such as:

You are alerting on a list of domains on your network, and don't want to set off thousand of alerts when someone accidentally adds "windowsupdate.com" to the list
You are reviewing sandbox reports, and don't want to get common non-malicious domains back in your reports

I've recently extended the whitelist ThreatCrowd uses when sites are marked as malicious, following feedback that a number of domains had been mistakenly flagged as malicious by users.

This coincided with Alexa announcing they would stop publishing a commonly used whitelist - the top 1 million sites. Thankfully Alexa have changed their minds about discontinuing the data-set, for now at least, and there are other similiar sources too.

Sources like this aren't well suited to matching against network data though - sites that are programatically accessed (eg; download.windowsupdate.com) often won't be listed in datasets designed to record human traffic. A better choice may be to use the top x domains on your network. However that does require access to network logs of a large network.

For this use case - I've used logs from networks that are publicly available online. There are plenty of people who (perhaps inadvertently) publish this online. In this case I've used data from freedom of information requests for the top sites requested on a number of UK government networks.

Two things to note are:

This data is biased towards the UK
I'd suggest only using domains seen on more than one network. For example one of the domains seen on only one network below is likely Chinese APT (yes, they're aware).

You can find the list below, for all your whitelisting needs:

Clustering the Threat Landscape

Much of threat intelligence is grouping together information to identify common traits in attackers.
To that end, I wrote a quick python script to identify common indicators in reports in Alienvault's OTX platform. You can see the output of this script in the image below, with some of the more interesting clusters annotated:

This isn't a perfect method - there are some odd links there that I wouldn't expect to see. But there are also some very interesting overlaps highlighted between disparate clusters of attacks that identify possible links between groups.

You can download and browse through the Maltego file [here] - and some of the clusters are displayed below.
Update: You can download the source file [here], to see what indicators reports overlap on. It's trimmed to the first indicator for each overlap.

BlackEnergy

Carbanak with a report on more commodity malware connected via the domain trader562[.]com

Lots of overlaps with Chinese APT

RocketKitten

Sony Attacks

Sunday, 28 February 2016

Crowdsourced feeds from ThreatCrowd

Voting

Voting was added to ThreatCrowd recently, and I've been pleased to see a number of users regularly contributing votes.

These votes provide a useful source of malicious indicators, and so I've now put these into a feed in two files:

https://www.threatcrowd.org/feeds/domains.txt
https://www.threatcrowd.org/feeds/ips.txt
https://www.threatcrowd.org/feeds/hashes.txt

These feeds are not a substitute for the scale of auto-extracted command and control domains or the quality of some commercially provided feeds. But crowd-sourcing does go some way towards the quick sharing of threat intelligence between the community.

Updates
These files are updated once per hour, on the hour.

API
You can submit votes via the interface, or a simple API:

This will place a vote for "good.com" being non-malicious:
https://www.threatcrowd.org/vote.php?vote=1&value=good.com

This will place a vote for "bad.com" being malicious:
https://www.threatcrowd.org/vote.php?vote=0&value=bad.com

License
This data is available for free, and commercial use is allowed. It's licensed under http://www.apache.org/licenses/LICENSE-2.0
I make no guarantees to the quality of the data.

Sunday, 4 October 2015

New Feature- Monitoring Infrastructure with ThreatCrowd RSS feeds

Today I've added RSS feeds to ThreatCrowd. These provide a simple way for monitoring attacker infrastructure.

For example - say your organisation has a significant footprint in Russia. You may be interested in attacks referred by ESET as "Roaming Tiger".

You can now monitor infrastructure of attackers using RSS feeds, for example-

This provides you with a simple way of monitoring when an attacker

Registers a new domain
Points a new domain at a server
A new malware sample is uploaded to an online sandbox

The power here is that you can combine multiple RSS feeds into a single RSS feed per actor to get a clean feed of activity. For example this feed combines the three above-

http://www.rssmix.com/u/8156326/rss.xml

Two caveats here are that:

Exp/20120158-A is used by multiple groups
As with any online research, consider the OPSEC implications. I wouldn't recommend using an online tool such as RSSmix for this

Monday, 10 August 2015

Example Threat: Dark Hotel 2

Kaspersky today released a report on updated attacks by a group they term Dark Hotel, previously best known for it's use of malware named Tapaoux.

Image from: https://apt.securelist.com/

You can browse some of this infrastructure within ThreatCrowd at:

Sunday, 5 July 2015

The Search Engine for Threats now supports Search

The tagline for ThreatCrowd is "The search engine for threats". Whilst it's great to see some people start to use this to describe the site, until now the search function has been sorely lacking.

Today I'm happy to release the first version of a real search function. Rather than just searching for artefacts such as domains or IP addresses, you can now search for more general terms such as organisation names.

An example
Below is a search for tibet - a theme commonly employed in targeted attacks.

There are a number of results here. The first, tibet.my03[.]com, is referenced as associated with a malware report, malware and is a dynamic domain (often employed by malware).

You may be wondering why this result came first.

Results are ranked by a number of factors, such an obvious PageRank/TrustRank style "number of hops from known bad" but also by a number of crafted rules. You can expect the see the ranking of results fluctuate as alternate ranking functions ate tested.

What next?
The implementation of a search function means ThreatCrowd has finally met the initial roadmap of core functions. Development now will be on incremental improvements to the interface to make research easier, and increasing the scope of data.

If you've got any thoughts on how to improve ThreatCrowd, or just general comments - I'd love to hear them.