SIRAcon Attendees, Start Your Engines

Posted on 25. Oct, 2013 by in Data Analysis, Data Science, Event, Industry, Metrics, Remediation, Security Management, Vulnerability Intelligence, Vulnerability Management

“Information is the oil of the 21st century, and analytics is the combustion engine.” -  Peter Sondergaard, SVP Gartner

SIRAcon

This week I attended SIRAcon in Seattle, a conference hosted by the Society of Information Risk Analysts. I spoke about the methodology behind Risk I/O’s “fix what matters” approach to vulnerability management, and how we use live vulnerability and real-time breach data to build the model, as well as why such a model performs better than existing CVSS-based risk rankings. However, there were a few persistent themes between the many qualified and excellent speakers at the conference. Combining and implementing these practices is not a simple matter, but organizations should take note, and as an industry, information security can evolve.

1. This is not our first rodeo.

Risks are everywhere – and other industries not that different from ours have caught on. Ally Miller’s morning keynote discussed the structured, quantified way in which fraud detection teams are built: starting with real-time data collection, updating of large, global models which guide decisions about fraud, and the ability to make those decisions in real-time. This requires clever interfacing with business processes and excellent infrastructure, but it’s been done before, and needs to be done with respect to vulnerability management as well. Alex Hutton used Nate Silver’s latest book on Bayesian modeling to raise some parallel questions about infosec. He drew analogues to seismology and counter-terrorism, and the maturity and relative similarity of those fields (large, often hard to quantify or observe risk) is something for us to explore as well. Lastly, his talk raised a healthy discussion on the differences between forecasting and prediction. A prediction describes the expectation of a specific event (“it will rain tomorrow”), whereas a forecast is more general, and describes the probability of a number of events over time (“there will be 2 inches of rain in december”, “there is a 20% chance of rain over the next day”). Largely, the discussion was focused on how management perceives differences between the two. In seismology, we fail at prediction because the mechanics are obfuscated, and so we can only forecast. The same seems to be largely true of infosec.

2. Good models need good data.

Adam Shostack from Microsoft gave a very convincing closing keynote on the value of data-driven security programs. Running experiments targeted at collecting data will generate scientific tools and take the qualitative (read: fuzzy) decision-making out of risk management. The alternative is the status quo – reliance on policies or measuring organizational performance against standards, which is paramount to stagnation, which no one can say about our adversaries. He states that although almost all organizations have been breached, it is incredibly difficult to develop models of breaches, largely because global breach datasets are hard to come by. Not so! We’re hard at work incorporating new sources of breach data into Risk I/O – but he’s most certainly correct that this is a hard project for any single company to undertake. Adam concluded with a call for organizations to encourage better sharing of data (hear, hear), and this mirrored the sentiment of other talks (particularly Jeff Lowder’s discussion of why we need to collect data to establish base-rate probabilities) about the need for a centralized, CDC-like body for infosec data.

So let’s get some data. We’re already off to a pretty good start.

Introducing the Risk Meter

Posted on 08. Oct, 2013 by in Data Analysis, Data Science, Feature Release, Launch, Metrics, Threats and Attacks, Vulnerability Intelligence

Risk Meter

You may have noticed we’ve been publishing a lot of information lately on what factors go into the likelihood of a successful exploit. Our presentation at BSidesLV and subsequent events touched on some of the work we’ve been doing based on our processing of over a million successful breaches we have observed across the internet. While this data continues to grow we’ve been able to glean some great insights into what factors matter most when making remediation decisions.

Our data-driven approach is leaving us pretty excited about our latest feature to hit Risk I/O, the Risk Meter. The Risk Meter takes a number of factors into account including: Exploit Analytics, Asset Prioritization, and Adjusted CVSS. Take a look at the Risk Meter FAQ for more information on rankings.

Because the Risk Meter is an asset-driven model, you’ll naturally find it in the Assets tab. As you apply filter and search criteria to your assets, the Risk Meter score will change to reflect the current group of assets you are viewing.

Want to see a patch report that reflects those assets and Risk Meter? Just click the patch report button directly beneath your meter.

Want to save the Risk Meter for that asset group to your dashboard? Click the save button and it will automatically save that group into your Risk Meter dashboard.

Risk Meter Dashboard

The Risk Meter dashboard gives your team and management a quick, at-a-glance view of your vulnerability and exploit risk across your business, categorized by what’s meaningful to you. Each Risk Meter in your dashboard will display your current real-time score as well as summarize each one with information on the number of vulnerabilities and assets, how many vulnerabilities are easily exploitable, how many are being observed as breaches in the wild, how many are popular targets, as well as the number that have been prioritized.

If you already have a Risk I/O account, feel free to check it out now in your Assets tab. Don’t have an account? Sign up for a free trial.

Stop Fixing All The Things – Our BSidesLV Talk

Posted on 06. Aug, 2013 by in Big Data, Data Analysis, Data Science, Event, Metrics, Risk I/O, Threats and Attacks

Last week at BSidesLV, Ed Bellis and I presented our view on how vulnerability statistics should be done. We think it’s a different and useful approach to vulnerability assessments.

Our contention is that the definitions of vulnerabilities in NVD and OSVDB are just that – definitions. As security practitioners, we care about which vulnerabilities matter. Much like looking at a dictionary doesn’t tell you anything about how often people use a common word such as “the,” a vulnerability database tells one nothing about which vulnerability trends matter. We solve this problem by using live, currently open vulnerabilities to do our assessments.

Since the slides are linked below and the recording is available here, in this blog post I want to give the quick-and-dirty summary without the theatrics, as well as point in some other interesting directions that this type of work could go.

What we had to work with:

1. 23,000,000 live vulnerabilities across 1,000,000 real assets, which belong to 9,500 clients.

2. 1,500,000 real breaches which occurred between June and July of 2013 and were correlated to 103 CVEs.

What we did:

1. Treated the two samples of breaches and vulnerabilities (incorrectly, but usefully) as coming from the same population.

2. Calculated the conditional probabilities of certain types of vulnerabilities being live-breached vulnerabilities.

Why it’s important:

While not statistically accurate, this is a great way to compare two policies for remediating vulnerabilities. Should you fix only CVSS 10? Or 9? Maybe you should patch everything. We try to answer the age old question: if you have $100, and really care about information security, how do you spend it?

What we found:

1. The best policy was fixing vulnerabilities with entires in both Metasploit and Exploit DB, yielding about a 30% success rate, or 9x better than anything CVSS gets to, and 15x better than random.

2. Randomly picking vulnerabilities gives one about a 2% chance of remediating a truly critical (that is, one that has observed breaches in the past two months) vulnerability.

3. Randomly remediating a CVSS 10 vulnerability gives you a 3.5% chance of fixing a critical vulnerability.

4. If your policy is one of fixing vulnerabilities in Exploit DB, you have a 13% chance of remediating vulns with observed breaches.

5. Metasploit only? 25% chance.

Here’s how it looks on paper:

Screen Shot 2013-08-06 at 1.34.54 PM

What else we should all think about given this new information:

1. SushiDude and Attrition.org gave a cool talk at BlackHat about vulnerability statistics bias. A lot of it had to do with how people use definitions. What does everyone think about how bias in vulnerability databases affects our approach of using live vulnerabilities for statistics? My contention is that the bias is mitigated quite well. As a thought experiment, imagine a perfect vulnerability database, with every vulnerability ever discovered and no bias. Would looking at that vulnerability database help you decide what you should remediate next?

2. Frank Artes from NSS labs had an excellent talk at BsidesLV about which vulnerabilities blow past IDS and 2nd gen firewall systems. It was interesting to hear that 26% of metasploits go undetected by IDS systems. How can live breach statistics account for those undetected metasploit modules or worse, blackhat exploit kits? What can live vulnerability statistics do to increase vendor awareness about the missing metasploits (a hashtag if I’ve ever heard of one)? We hope to be able to illustrate which of the missing ones are top priority for _vendors_ to include in their detection packages based on the prevalence of live vulnerabilities susceptible to such exploits in the wild.

As always, your thoughts and feedback are very welcome. Stay tuned for more on breach/vulnerability correlation and the insights we glean from Risk I/O data. And you can follow us on Twitter @RiskIO.

The Role of Security Mapping in Vulnerability Management

Posted on 07. Feb, 2013 by in Data Analysis, Guest Blogger, Industry, Precognition, Vulnerability Management

Increasingly, security management organizations are coming to rely on a unique type of geography to recognize where threats and vulnerabilities are active, and where security exploits are occurring. The geography in question maps fairly closely to the physical map of the world. Because Internet links that connect sites and users to service providers are involved, along with prevailing local Internet topologies between the edges of that global network and local elements of its core, this geography tends to be more compressed and to be subject to strange or interesting hops between locations. Of course, this reflects the peering partners at various points of presence for SONET and other high-speed infrastructures, and doesn’t always reflect the same kind of geographical proximity you might see on a country or continental map.

Nevertheless, keeping track of where threats and vulnerabilities are occurring is incredibly useful. By following lines of “Internet topography” spikes in detection (which indicate upward trends in proliferation, or frequency of attack) are useful in prioritizing threats based on location. For one thing, networks that are geographically nearby in the Internet topography are more likely to get exposed to such threats, so it makes sense to use this kind of proximity to escalate risk assessments of exposure. For another thing, traffic patterns for attacks and threats tend to follow other typical traffic patterns, so increasing theat or vulnerability profiles can also help to drive all kinds of predictive analytics as well.

It’s always interesting to look at real-time threat maps  or network “weather reports” from various sources to see where issues may be cropping up and how fast they’re spreading. Akamai’s Real-Time Web Monitor provides an excellent and visually interesting portrayal of this kind of monitoring and analysis at work. In the following screen capture for example, we see a handful of US States where attacks have been detected in the last 24 hours.

Security Mapping

Akamai’s “Real-Time Web Monitor” Displaying a Security Map

In general, threat, vulnerability and attack mapping work well because such data makes for intelligible and compelling visual displays. Human viewers are familiar with maps, and quickly learn how to develop an intuitive sense for threat priority or urgency based on proximity and the nature of the threats involved. That’s why so many security service providers use maps to help inform security administrators about safety and security in their neighborhoods, and around the planet.

About the Author: Ed Tittel is a full-time freelance writer and researcher who covers information security, markup languages, and Windows operating systems. A regular contributor to numerous TechTarget websites, Tom’s IT Pro, and PearsonITCertification.com, and UpperTraining.com, Ed also blogs on Windows Enterprise Desktop and IT Career topics. His latest book is Unified Threat Management For Dummies. Learn more about or contact Ed at his website.

Playing Around with Game Theory: Smart Data > Big Data

Posted on 06. Feb, 2013 by in Big Data, Data Analysis, Industry, Precognition, Risk I/O, Vulnerability Intelligence

There’s been a lot of talk about Big Data in the security space over the past couple of years, and it seems that almost every week a new Big Data offering enters the space, whether it’s in discussion, in development, or in production. It’s no secret that here at Risk I/O, we’ve embraced the industry’s demands and are hard at work developing our precognition offerings, many of which have been dubbed “Big Data.” This is accurate, since we aggregate and correlate data both inter- and intra-client.

A Game Theory Model

A Game Theory Model

However, in traditional Big Data problems—such as consumer shopping or health care—we would be looking to mine huge data sources for patterns of behavior, or commonalities between clients and attackers. In information security, we have huge amounts of SIEM data on activity around our assets, and great data on which firms have which vulnerabilities. The problem is that we’re missing successful exploit data or breach data, so just hunting for patterns or casting regressions isn’t quite as productive, because what we’re after are the probabilities and locations of attacks. Given the limited exploit data out there, I’ve been looking for some workarounds. Thankfully, there are some alternative ways of reducing risk.

I wanted to share a quick insight we’ve gathered in the process of ramping up new offerings. Particularly, I’ve invested some time into analyzing game theory as a tool for infosec analysis. Game theory is an applied field of mathematics developed as a means of computing optimal strategy sets for rational actors.

A common application is to analyze adversaries locked in a repeated game, with the outcome of one affecting constraints on the subsequent games. Repeated games assume that players will have to take into account the impact of their current strategy on the future actions of other players; this is sometimes called their reputation. In web security, the analogue is clear cut: hackers are looking to choose which assets to exploit, and each one of us wants to decide which assets to patch first, which vulnerabilities to deal with first, etc. For a good literature review of the field, take a look at Game Theory Meets Network Security and Privacy. Good work is being done in academia already on the subject, and the results allow us to turn Big Data problems into tractable “medium data” problems rather quickly:

Asset Network Topology Might Matter Less than You Think

A recent paper by Sandia National Laboratories and Duke University uses a game theoretic model of arbitrary network topology, which varies in both the topology and the degree of uncertainty in links. It’s a great use of game theory to determine optimal strategies.  The main result is that:

IF 

(Some assets are prioritized higher than others, that is, utility from defending the network is non-homogenous)&&(the costs of defending or remediating an asset’s vulnerabilities are not too high)

THEN 

Network topology doesn’t affect the payoff to the defender.

Squirrel lifting big data

Making big data lighter.

Most real world systems meet the first condition, and while “high costs” are a judgment call (there is some evidence in this blog post), they are minuscule in comparison to the costs of a data breach. The result is profound: it means a map of network topology—or “second order” spillover impacts from one asset being compromised—don’t need to be factored into a data model when choosing which assets to defend. All of a sudden, the data looks smaller. This does not imply that we should ignore the topology of networks, but it does mean that we need to think critically (test, test, test) about whether including all the data we have is the best approach.

The next question we need to answer is – does result this make sense? After a good discussion with @sintixerr and @MrMeritology on twitter (started by this post), I’m convinced that it does. Here’s why network topology likely doesn’t matter when defining the defender’s optimal strategy.

  1. Attack Paths - In much of game theory, the best defense against an attacker, especially one about which there is little information, is dynamic. The reason? Attackers randomize their attack patterns and paths. As the number of attack paths grows, the value of a particular asset itself matters more than its’ effect on others. Simply put, if they’re going to get there, hackers will get where they need to eventually.
  2. Attacker Types and Evolution - The importance of assets changes (see @sintixerr’s post above) as a result of evolving hacker strategies (and types of attackers). And since we can’t (yet) predict how this importance will change, we also can’t predict which links in a network will matter more than others. It’s important to note here that most enterprises are also threatened by more than just one type of attacker, so any risk assessment will have conflicting estimates of risk. The aforementioned game theory paper proves this point by showing invariance to attacker type.

There’s too much uncertainty about if, who, where, or when one will be attacked. Credit to @MrMeritology for this Smithonian article, which distinguishes the problem at hand: “A mystery cannot be answered; it can only be framed, by identifying the critical factors and applying some sense of how they have interacted in the past and might interact in the future.”  The takeaway here is that Big Data is not always smart data. Big Data will let us solve every jigsaw puzzle and an NxN rubrik’s cube. Smart data will tell us which factors truly matter.

Before we launch into full-scale Hadoop implementations and start firing up R regressions on every variable we can get our hands on, it’s worth our time to take a step back, think about what’s available to us, what’s not, and what that means. My contention is that thinking about optimal strategies, which are robust to uncertainly, can alleviate the need to predict exploits – at least until the data gets big enough. More insights from the frontlines coming soon.

Heads Up! (Display)

Posted on 22. Jan, 2013 by in Data Analysis, Feature Release, Metrics, Risk I/O, Vulnerability Management

Heads Up Display

Visualizing your vulnerability data with our new Heads Up Display.

I’m happy to share our latest enhancement to visualizing your vulnerability data. Today, we are launching a new Heads-Up Display (HUD): a “mini dashboard” if you will,  that allows you to visualize the current state of your vulnerabilities and defects.

Our new Heads-Up Display shows a live presentation of your vulnerabilities. It provides up-to-the-minute information on aspects of your vulnerability management program such as scoring, asset priorities, exploitability and impact calculations, with more metrics on the way. Each metric is interactive: rolling over a graph will show you the actual value of the attribute represented in that graph, while clicking a graph performs a live filter based on that attribute.

A simple use case for quickly finding vulnerabilities in your environment that have a very high likelihood of being exploited may be as follows:

  1. Navigate to the Vulnerabilities tab within your instance of Risk I/O where you will find the Heads Up Display.
  2. Let’s start by filtering on vulnerabilities that don’t require local access by clicking the Network portion of the Access Vector chart. This filter brings our open vulnerabilities down from 63,060 to 50,970.
  3. Now lets drill down further by continuing to look at the simplest of vulnerabilities to exploit. I’ll click the Low value for Access Complexity and None Required for level of Authentication. This brings our list down even further to 16,462.
  4. From here I can narrow the number of vulnerabilities to tackle by filtering on their impact subscores. This will give us issues that are not only likely to be exploited but with higher impacts. Lets filter by choosing Complete impacts of Confidentiality, Integrity and Availability. This brings our open list down to 5,709 open vulnerabilities.
  5. Next up, we’ll take our current list and narrow it further by only looking at vulnerabilities that have a Known Exploit. Choosing this value to filter on results in 1,111 open vulnerabilities.
  6.  Heads-Up Display

    The 16 most egregious vulnerabilities via HUD.

  7. So far, we’ve been focusing on vulnerabilities that are easy to exploit, could have a higher impact on our environment, and have a publicly available exploit from sources like the Metasploit Framework, ExploitDB, etc. Let’s take this one step further and filter only on vulnerabilities within our DMZ that may be publicly facing. Since I have tagged these assets with DMZ within my instance of Risk I/O, I can simply select the tag ‘DMZ’ to filter on. This gives me a very short list of 16 open vulnerabilities to work with.

As mentioned, this is a simple use case to find the most egregious of vulnerabilities within our environment and I’m certain you will have and find many others. We think HUD will be one of the easiest ways to stay on top of your vulnerability management. We’ll be sprinkling more mini visualizations throughout Risk I/O in the future as we identify specific metrics that would be helpful to see in a more visual fashion. Give it a try and let us know what you think.

Mario Mendoza Goes to SecTor

Posted on 05. Oct, 2012 by in Data Analysis, Event

Mario Mendoza set what would become known as The Mendoza Line

I wanted to thank the folks at SecTor for having me out and allowing me to bring the Security Mendoza Line discussion north of the border. As last year, I had a great time and I’m always impressed with how well the conference is run. This was also the first year we did a sponsorship and we brought along the new t-shirts to share. Watch for the ‘shirt-o-matic’ coming soon.

I was able to meet a ton of people and the kind folks at the Liquid Matrix Security Podcast allowed me to ramble on a bit as part of their ongoing coverage of the con. If you haven’t already, go subscribe to the podcast as they have already been piling up good content. Below is my deck from the talk. It was the first time I gave this talk and after a few follow ups afterwards, I will likely be making some modifications. I also have some additional data to add to it based on real world vulnerabilities and how they relate to the Mendoza line.

After the talk, I do have to agree with both Alex Hutton and Josh Corman that HD Moore’s law resonates more with our audience. Perhaps I could create a hockey analogy for the next time I’m up at SecTor…

Joining the Data Revolution

Posted on 22. Aug, 2012 by in Award, Big Data, Data Analysis, Event, Industry

Here at Risk I/O, we’re really big fans of data. Given the right data you can make insightful business decisions very quickly. This is one of the core values we build into every feature release.

DataWeek Logo

Risk I/O is excited to be the recipient of a DataWeek Award as a Top Innovator in the Security/e-Governance category.

With our data-driven approach to security, we’re excited to have been selected by the DataWeek Awards as a Top Innovator in the Security/e-Governance category. This is the first annual DataWeek event, which is used as a platform for data-centric
companies to discuss how big data can be harnessed. Risk I/O is honored to be joining a long list of innovative companies in accepting a DataWeek Award.

Throughout the remainder of 2012, we’ll be releasing data-centric features that will aid our users in decision support. Within the security community, we have seen a real issue with managing the mountain of data produced by vulnerability assessments, penetration testing and static analysis. There are very specific use cases that we are building into our product which include prioritization, predictive analytics, and an aggregated vulnerability database. We want our users to understand what’s the most important vulnerability to fix first and how vulnerable their data is to a specific threat. We also want our users to be able to easily access all available information on a specific vulnerability right in our product. We’ll be informing our users when these features are available.

We’re excited to be part of the Data Revolution and to be recognized for our commitment to it. Now to get back to building out some data-centric features for our users, fast and furious!

Hitting Above the Security Mendoza Line

Posted on 14. Aug, 2012 by in Data Analysis, Feature Release, Penetration Testing, Vulnerability Management

Risk I/O can now be used to identify publicly available exploits to your existing vulnerabilities. Our development team has made it possible for Risk I/O to match attack vectors from databases of quality assured exploits, such as Metasploit and ExploitDB, to applicable vulnerabilities. This information, paired with vulnerability data from assessment tools, allows you to understand how your organization is vulnerable to attacks.

Risk I/O's known exploit feature allows you to know how hackers could attack your organizational data.

You can now get information on publicly available exploits for a specific vulnerability.

In an earlier post, I wrote about the importance of focusing on data that allows you to “hit above the Security Mendoza line,” or the threats most likely to occur based on the evidence and ease of exploit. Alex Hutton referred to the Security Mendoza Line when talking about vulnerabilities exploitable with MetaSploit modules. Josh Corman expanded on this quite a bit with HD Moore’s Law. We built this feature to weed this out of your environment and allow you to hit above the Mendoza line .

We often talk about “enough security” and Josh frames it correctly by stating this is InfoSec table stakes. By combining data from these publicly available exploits and network accessibility, you can truly identify some low hanging fruit and protect yourself from the Point, Click & Pwn of the most casual and opportunistic attackers.

For the dataheads in the audience, a quick and early glance suggests about 14% of active and open vulnerabilities have publicly available exploits through one of these tools or databases. I think there is more interesting views to be had, but clearly an indicator that we have a long way to go before we can protect ourselves from even the most casual adversaries.

If you are a current customer, you can access this new feature in the Vulnerabilities tab. Don’t have an account? Signup for our free version!

Security Intelligence != SIEM

Posted on 05. Mar, 2012 by in Data Analysis, Event, Metrics, Security Management

I’ve just returned from RSA, BSides and Metricon and thought I would pen a few of my thoughts while they’re still fresh in my mind.

On Monday I had the privilege of participating in a panel on Data Driven Security at Metricon 6.5. Scott Crawford moderated and has a great blog series on data driven security. It was an interesting group of backgrounds between myself, Mark Clancy, Chris Eng, Micha Govshteyn and Martin McKeay. Some of the participants were further along in their analysis of the data they were collecting while some had a volume of data (Akamai) most of us could only dream of. While I don’t think there was anything too surprising that transformed from the panel, the final question from Russell Thomas was the exception. Russell asked, and I’m paraphrasing, if we had one new open job req we could hire for right now, how many of us would hire someone dedicated to this topic. What surprised me the most wasn’t the question but roughly half the panel including myself answered yes.

We are very dedicated to data-driven security and have very specific use cases that we are building into our product. I went in to Metricon worried that security intelligence would be more talking about SIEM or the various ‘State of the Industry’ reports published by several security vendors. While I don’t have an issue with those solutions and actually appreciate several of the published reports, utilizing big data in security can and should go well beyond these. One useful way we see to use this data as part of a decisioning system is through prioritization.

For example: if I have one million identified security risks, realistically I have little chance of remediating everything. How do I decide what’s most important? As a service provider that has visibility across many organizations, we can take into account a lot of different factors to help determine when someone becomes a ‘target of opportunity’ including postures across these issues as well as threat data across both public and private networks. This, of course, is one of MANY examples on how you can feed security and operational data into a system that helps make smarter security decisions.

While I was pleasantly surprised to not be talking about SIEM at Metricon, walking the show floor at RSA brought all my fears back and then some. Big Data was a huge topic at RSA but as mentioned by the guys at Securosis, there was a lot of repurposing of existing products like SIEM. As I mentioned to a coworker as we walked the floor, “it’s like Vegas minus the fun.”

RSA aside, there are plenty of examples out there of using large amounts of data in the real world to aid decisioning systems. One shining example is what Preston Wood and the team over at Zions Bancorporation are doing. They have taken security intelligence way beyond SIEM. Imagine taking these capabilities and setting them atop data that Akamai or other large service providers may house. Additionally, if you haven’t seen the post by Ben Sapiro yet, go check it out. While it’s not focused on big data in security, he talks a lot about analytics and at the end of the post lists a LOT of real world metrics anyone can start with to improve their program.