Last week I had the opportunity to present at the Best Practices for Technology Symposium. I have to be honest, I’ve never heard of this event and given the name it’s easily missed. In fact, given my recent post on “best practices” and vanity metrics I would have likely avoided an event with such a name. But that would have been a mistake.

Gene Kim introduced me to Fred Palmer who runs this event which is why I seriously considered it. It turns out it’s nothing like what I thought but rather more than two days of emerging technology companies presenting some of their latest tech. I only wish I knew about this event earlier and I wish that I’d known the format was so open. It’s refreshing to hear an audience that actually wants to talk about whether or not the solution you’re pitching works for them rather than the thinly veiled sales pitch cloaked as thought leadership. Having been a practitioner the majority of my career, I think this format is sorely needed. Fred has done a great job bringing together interesting security technologies and providing open and honest feedback. Much like IANS, it has a great workshop format in an informal setting.

One question during the workshop really resonated with me:

Do you think organizations know what’s important to them?

Of course, the sad truth is “it depends.” We were talking about creating a platform like Risk I/O that was created with flexibility in mind. This allows users to slice the data into views that are important to them in order to get better and faster insight. But this is a valid question. What if the organization isn’t sure where to start or what should be important to them? We like to think our priority and trending along with the Heads Up Display are a great start but we will continue to help our customers by flagging and alerting on issues we see as important while maintaining transparency and flexibility. These go well beyond the standard CVSS calculators and take into account real in-the-wild information.

A common question we get is “What are the metrics others are using to measure themselves against?” We will continue to share important metrics to help teams jump-start their programs. It’s great to see practitioners getting together and sharing information that will benefit and raise the bar, and we’ll continue on our mission in helping you gain visibility into what’s important.

Our SmartSearch feature has gotten, well, even smarter. You already know that with SmartSearch you can choose many fields from many criteria at once, enabling you to filter down to only the vulnerabilities or assets you need. Well now you can save the SmartSearch(es) you perform on your vulnerabilities and assets in Risk I/O for reference later.

Saving a vulnerability or asset search is easy. Simply perform a text search, change a numeric slider or select a facet, and a “Save” button will appear at the top of the Filters sidebar. Click that button, and a [NAME] text input will appear above. The name will auto-populate from your text search, if there is one, otherwise you can provide your own, then hit “Save”. Note that searches are saved per-user, so your colleagues can create their own searches as needed.

Performing a text search, changing a numeric slider or selecting a facet on the Vulnerability Table or Asset Table can now be saved for future reference.

When you want to reference the search later, just select one of the searches located within the “Saved Search” section. Searches cannot be updated once saved; but if you make modifications to a saved search, you can simply save your search with a new name.

Searches can also be deleted. Simply hover over the search name and select the trash can icon, and your saved search will be deleted.

Interested in taking this feature for a spin? Simply visit the Vulnerabilities or Assets tab and start filtering your data. Not yet a Risk I/O user? Visit our website to signup for a 30-day trial of our Pro plan.

A Tale of Two Uncertainties

There are fields where precision is of the utmost importance. In fields of exploration (physics, chemistry, arguably mathematics), we attempt to seek out the truths of the world around us, to get better and better models of what’s going on. In fields of manufacturing (chocolate making, farming, engine casting) precision matters because it produces better products.

What do these fields have in common? From where I sit, behind the terminal and on top of a wall of statistics, the least common denominator is that they have well defined basic data units. Mathematics has axioms and proven theorems, Chemistry has the periodic table, Physics has quarks, chocolate has cocoa beans, and engine casting has alloys and sandmolds.

Information security is nowhere close to those. Uncertainly lurks around every corner, it is inherent in every problem. Because at its core infosec is a game between attackers and defenders, and information is always intentionally imperfect. In this post I will discuss a far more terrifying type of uncertainty – data definitions.

Data Fundamentalism

Data analysis is a language. At it’s most useful it is a way to communicate complex findings to those who don’t have time, skills, or access to the same information that the analyst does. And much like any language it requires static, basic definitions. The first problem is well known: our definitions are uncertain and sometimes late to the game. The second is worse: we don’t understand the source(s) of the uncertainty.

For anyone working on vulnerability management or predictions (excuse the lack of ‘Big Data’ usage here), this second problem is huge.  As @katecrawford writes in her recent HBR article The Hidden Biases in Big Data, “data fundamentalism” has become a pervasive problem. This is the notion that predictive analytics and well-crafted correlations reflect the objective truth. Here’s the relevant sample:

CVE is NOT the Periodic Table of Elements

I work with vulnerabilities. There are a few places that define vulnerabilities, but CVE is the most universal set of definitions that I have to work with. Yet thinking of CVEs as elements on the periodic table is a grave mistake. Before creating synthetic polymers (read: useful analytics) out of these elements, we need to understand the biases and sources of uncertainty in the definitions themselves.

For example, take a look at this finding from a research team at Concordia University in their 2011 paper Trend Analysis of the CVE for Software Vulnerability Management:

There are many such papers out there; it is frightening to think they might guide organizational decision-making. This type of analysis misses the boat on what is being analyzed; it takes CVE to be analogous to the Constitution for legal scholars. An increase or decrease in their frequency, or the types that are being published over a time bucket can have wildly varying biases.

Let’s dive into some of them. The aforementioned HBR article also alludes to this: there’s no such thing as raw data. People working in data today need to take a clue from the less quantitative disciplines and take a look at where the data originates and how it’s gathered.

A Brief History of the Time: From @SushiDude to Today

CVE is a dictionary of known infosec vulnerabilities and exposures, and it is intended as a baseline index for assessing the coverage of tools. It is not intended as a baseline index for the state of infosec, as the aforementioned paper takes it.

Let’s start with this: the Wikipedia page is dead wrong. At this year’s RSA, I wanted to delve deeper into the exact process of CVE creation, and sought out @SushiDude, the father of CVE. Here’s the story:

Looking at the volume of CVEs seems to suggest that steadily increasing CVE disclosures mean ‘the state of security is getting worse, or some such poorly structured inference.

However, this is not a dictionary. This is a company with limited resources, attempting to streamline a process. This is easily seen when looking at the rate of disclosures over time. Note the changes in process cause a reduction in throughput first, then an increase. (Actually, this leads me to believe there was a change in the process in 2011 as well)

Their objective, from my conversation with them, is to increase throughput. This makes perfect sense – inform the public about as many standardized vulnerabilities, as their resources will allow. However, in this process lies the essential biases inherent in the basic units of vulnerabilities:

1.     Descriptions – Some vulnerabilities are inherently more difficult to describe. Some attack vectors entail chaining of two, three, even five different weaknesses in an application. Analysts can publish five other vulnerabilities in the time it takes to write up a complicated chained one.

2.     Categorization – A CVE is meant to exist independently of the multiple perspectives on that vulnerability. In the submission process, a whitehat might find a vulnerability, a vendor might submit the same one, or a third party may discover it as well, all assessing it differently. The more of such sources, the harder it is for analysts to standardize this information. For another great argument about how the process of CVE categorization influences statistics, see this OSVDB post.

3.     Prioritization – There’s a vast and unexplored sea of vulnerabilities out there, and limited manpower. So how does MITRE decide which of them to look at? The advisory board helps made decisions about which vendors to prioritize, and some vulnerabilities get left in the backlog. A nice phrase employed to describe this sea of backlog is the “php.golf.” The opportunity costs of working on all of those are missing a Windows vulnerability, and so they stay put. In fact, there are a few CVE Numbering Authorities which get (rightfully so) preferential treatment. Also note how low severity vulnerabilities rarely enter the picture unless throughput is at high level (i.e. most of the high and medium severity stuff has been taken care of).

Here’s a little light reading to prove to you this isn’t make believe, from the CVE internal mailing lists:

And if you’re truly interested, dive deep into the link above to see all the inner workings of how a CVE comes to be. Regardless, it’s always good to know exactly what you’re working with.

As you may recall reading, our development team has been busy over the last few weeks rolling out new features that will make it even easier to manage and monitor your vulnerabilities.We want to invite you to join us on Wednesday, April 17th at 2:00PM ET for a webinar given by Risk I/O CEO, Ed Bellis. Ed will provide an overview of these new features and functionality in Risk I/O, including:

1. Patch Reporting

2. Bulk Creation of Trouble Tickets

Using a quick search-and-select, you can now create tickets for hundreds of vulnerabilities and assets in a matter of seconds using our new bulk creation of tickets feature.

3. Filtered Dashboards

You can now filter dashboard metrics using the custom tags you’ve created right in Risk I/O. This empowers you to perform comparisons across your entire business and technology stack.

You can register for the Risk I/O webinar now. You’ll receive a free security t-shirt for attending (see below).

Not a Risk I/O user but would like to give our vulnerability intelligence platform a spin? You can signup for free!

This is a guest blog post by Jacques Benkoski, Risk I/O investor and Board member.

A complete, global transformation of the landscape in the security software domain is underway. Countless articles have been written about how we have moved beyond the casual attacker cooking a virus for fun and/or profit to something completely different and far more dangerous to the general public. Recently, we have seen the emergence of professional hacking groups, the industrialization of intrusions by organized crime and the chilling prospects of state actors penetrating the cyber world for espionage and counter-espionage.

Similar to the Aviation industry, there has been a complete transformation in the Security industry.

One hundred years ago the Wright brothers changed the world with the technological leap of the airplane, which started as a fantastic new toy for the rich, and quickly became a mass form of commercial transportation, that, when adopted by governments for the military, also radically changed the way in which wars are fought.

Similarly, cyber technologies have been transformed, moving from the drive-by shooting represented by the viruses of yore to the full-fledged equivalent of industrial wiretapping. And, with the appearance of government-based and often government-backed attacks, we have crossed into another realm.

There is no need to reiterate the highly publicized Stuxnet virus attack on Iran, as well as some of the lesser known damage that its mutating siblings have inflicted before and since then. The U.S. and several other countries no longer have only a cyber defense department, but have also assembled top teams in official cyber attack structures.

It is still hard to fathom on which cyber fields these wars will be fought but I have little doubt they will. One can imagine many of them being fought in the quasi-silence of air-conditioned mega data centers. But some of them could do real physical damage to our infrastructure, transportation systems and communications, thereby causing injuries and even death to innocent people. But even if that worst-case scenario can be avoided, it is not science fiction to imagine a small country having to surrender without much of an old-fashioned war, simply because its government has lost all control over its infrastructure, transportation and communication networks.

For some of us involved in the security software industry, we are already seeing a ramp in spending by government and industry alike. The threats are already real and the losses are far from imaginary. Industrial espionage is no longer the sad privilege of Fortune 500 companies and defense contractors; it is a daily occurrence all the way to small businesses. Government bodies all around the world are issuing RFPs looking for technology to protect their countries’ assets and utilities, and are scrambling to identify and patch their obvious cyber vulnerabilities. Just like with aviation last century, large companies will be built that will help manufacture, protect and attack, and this represents a huge opportunity for entrepreneurs to respond to market needs.

As citizens of the world, we can only hope that the forces will balance through some form of cyber deterrence, and that the conflicts will remain limited—with no other victims than a few bank accounts, some stolen contracts and an occasional out-of-control centrifuge.

About the Author: Jacques Benkoski is a partner at U.S. Venture Partners (USVP), as well as an investor and Board member of Risk I/O. He is focused on fostering USVP as an active investor in the Israeli market and in the IT markets in the U.S. Jacques is an avid competitive sailor, a skier, a swimmer and can often be seen on a motorcycle.

After recently reading a post from Gary McGraw at Cigital arguing for software security training, I became a bit frustrated with cited “evidence” and posted this out on Twitter and received a short follow up from Lindsey Smith over at Tripwire…

Now let me say upfront, I have a lot of respect for Gary and his work AND actually agree with him on the subject of software security training. I’ll get into the why I agree with him in a bit. That said, here’s where my frustration comes in. Gary references the BSIMM as evidence that software security training works. Evidence? I find the BSIMM interesting but it leaves the taste of vanity metrics in my mouth. For those of you not familiar with the term vanity metrics, Eric Ries talks about them a lot as part of The Lean Startup:

“Actionable metrics can lead to informed business decisions and subsequent action. These are in contrast to “vanity metrics” – measurements that give “the rosiest picture possible” but do not accurately reflect the key drivers of a business. Vanity metrics for one company may be actionable metrics for another. For example, a company specializing in creating web-based dashboards for financial markets might view the number of web page views per person as a vanity metric as their revenue is not based on number of page views. However, an online magazine with advertising would view web page views as a key metric as page views as directly correlated to revenue.”

I think the BSIMM and best practices within information security often fall under the definition of vanity metrics. There are things I like about the BSIMM and it’s a great start but only focuses on one half of the data. Telling me what many companies are doing for their security controls becomes a lot more interesting when you also tell me how those controls faired over time. I would love to see the BSIMM and other models like it evolve into an evidence-based set of controls. Today, they certainly should not be cited as evidence that any control within them works as we’re completely missing that side of the picture. This is also not a post to pick on BSIMM but rather an attempt to call out our industry citing best practices without evidence.

I mentioned earlier in this post that I actually agree with Gary on software security training. The reason I can say this is based on evidence, not best practices. At my former employer, we implemented a number of measurements around application defects and specifically security defects. We also did various software security training exercises both internally and with help. As part of this we measured things like defect rates and density within specific groups both before and after. We continued these measurements over time and saw material drops in most categories. Was it completely due to training? No, but we saw a measurable impact each time that correlated with a specific set of training. It’s evidence similar to that I’d like to see combined with a set of “Best Practices.” At best, best practices are a set of things that others *may* be doing; at worst they are meaningless vanity metrics.

Hot on the heels of our filtered dashboard  and patch reports feature releases, we’re announcing our latest security tool integration. Risk I/O can now integrate with the NTOSpider dynamic application security testing (DAST) solution.

You can now integrate NTOSpider directly into your instance of Risk I/O.

Adding NTOSpider to your selection of Risk I/O connectors allows you to leverage its unique capabilities to detect vulnerabilities within your applications. With NTOSpider, you can scan mobile and web applications that utilize technologies like REST, AJAX, JSON and GWT. NTOSpider can automatically crawl, detect and attack vulnerabilities that were previously only discovered via manual testing, and it works to eliminate both both false positives and false negatives from appearing in your scan data.

Setting up your NTOSpider connector in Risk I/O, like our other integrations, is easy and requires simply adding it to your Risk I/O instance via the Connectors tab. Not a Risk I/O user but would like to try out this integration? Signup for a free account!

Dashboards and metrics the way you like ‘em. Today we are launching our new filtered dashboards feature, which allows you to use custom tags to see different views of your vulnerability metrics.

In addition to the default dashboard view, which provides a summary of your security state across the entire organization, you can now filter those metrics using the custom tags you’ve created right in Risk I/O. This empowers you to perform comparisons across your entire business and technology stack.

Filter your dashboard by custom tags to see a specific view of your vulnerability metrics.

Simply select a custom tag from the dropdown to change your metrics view.

Simply select a tag from the drop-down list located above the dashboard. You’ll notice that your vulnerability metrics will update depending on the tag chosen. Selecting a custom tag on one tab will automatically cause the metrics to reflect that selection when navigating to another tab. You can reset your search by selecting the “x” located in the drop-down. Please note that custom tags can be managed right on your Vulnerabilities and Assets tabs. Because the Benchmark tab aggregates data our global data set, the tag filter is disabled on that tab.

We encourage you to check back often as we add additional metrics to the Risk I/O dashboard, as we anticipate the use of tags will give you some very unique views into your security posture.

The Risk I/O dev team has been developing features at a ridiculous pace with no signs of slowing down. We will be releasing a host of new functionality to our vulnerability intelligence platform over the weeks to come, so stay tuned. Our latest additions will help you identify patches that will reduce the most amount of risk across your environment and quickly push them out to your ticketing system or manage remediation directly within Risk I/O.

We know that identifying remediation can be tedious, so we set out to solve this in a simple way. Patch data is pulled in directly from multiple sources and now made available via patch reports on your Assets tab. Viewing the patch report will give you a quick view of un-patched systems grouped together by patch and sorted in order of total risk score. This view gives you the biggest “bang-for-your-buck,” (in other words reducing the most amount of vulnerabilities with the least amount of work).

View the patches that reduce the most risk via our patch reports.

Creating trouble tickets in Risk I/O has always been fast and simple through our integration with ticketing systems. But we’ve now made this even faster by adding bulk creation of tickets in Risk I/O for both vulnerabilities and assets. With a quick search and select, you can create tickets for hundreds of vulnerabilities and assets in a matter of seconds. Within the vulnerabilities tab you can send multiple vulnerabilities to a single ticket via the bulk ticketing feature, or within the patch report you can create a single ticket to patch thousands of assets at once.

Send multiple vulnerabilities or patch multiple assets with a single ticket.

Log into your account now and try these features for yourself. Of course, if you don’t have a Risk I/O account yet, you can signup for free.

This was my first time attending RSA, and on top of that I am fairly new to the Security industry. If RSA were a Senate race, I would be Ashley Judd. I am not, however, new to statistics. The following is an outsider’s perspective on Metricon, one without any preconceptions of the space. Spoiler: to be more secure as an industry, we need to share information. Below I’ll give you the why and just a little bit of the how.

Davids & Goliaths

From the numerous discussions I’ve had in the halls of RSA and at my Metricon table, the question of security team maturity has been the underlying force, shaping most of the discussions. Vendors decide which customers to go after based on team size and maturity, some choosing not to engage the small ones, some choosing to cater specifically to the little guys. Almost every work group at Metricon included at least one metric to measure the maturity of a security program, such as “% assets scanned,” “development lifecycle time,” “mean time to mitigate” or “time to discovery”.

In his lightning talk about setting up new security teams at Metricon, Dr. Chuvakin advocated for “one metric per broad domain,” these domains being identity management, vulnerability management, etc. Identifying a quick out-of-the-box way to set up a new security program is a terrific idea—it correctly highlights that the Security space is not just about improving existing programs, but equally as important—proliferating secure practices in “low” places. Metrics like “patching speed” and “% assets scanned” is a one or two man job for a smaller environment.

Pair maturity metrics with Dr. Chuvakin’s talk about the “minimum set of metrics” to get the ball rolling, and I’m faced with the age old question:“Does size really matter?”

A perfect world of information sharing.

My contention is that while the complexity or efficiency of a security program might scale with size, the way in which we measure effectiveness should not. In fact, in my perfect world riddled with unicorn lairs and information sharing, I could tell you how your security team stacks up against someone else’s: 38% from beyond the arc, 82% free throw, decent ticket sales, not too many lawsuits. However, I’m not implying that all security teams are created equal. Let’s give credit where credit is due – size helps.

Business.Ops.Security.Scale!

If you have perfect knowledge of your assets, the security team’s job is easier. Having good estimates of the dev time it takes to remediate certain issues on certain assets also simplifies the decision process. Business operations and reporting up should factor into security decisions. This implies something at the heart of many of the metrics I overheard at Metricon—scaling operational efficiency eases the strain on the security team, makes remediation faster and cheaper, and expands coverage. However, one can easily imagine an infosec team paired with a great operations team that’s not very good at choosing what to remediate.

“There’s nothing in life that you can’t improve by throwing money at it” – Congressman Dr. William Foster

Kaiser Permanente’s and Hubbard Street Research’s The Probability of Exploit talk at this year’s RSA is perhaps the most complex metrics practice I’ve seen out there. It involves training analysts to make bias-free decisions, running Monte Carlo simulations, machine learning from the outcomes of said simulations … you get the picture. Attempting to estimate the expected returns of remediation is a multi-analyst, multi-data scientist, dev-time intensive process. I have no doubt that this complexity pays off – the remediation decisions become prioritized by dollar amounts, easy to report up to management, etc. Getting back to my theme here—a bad machine learning coefficient here or there—and such a system would never know that it’s remediating the wrong vulnerabilities.

Security teams should strive to “hit above the Security Mendoza line,” or the threats most likely to occur based on the evidence and ease of exploit.

Our own Ed Bellis often talks about hitting above the Security Mendoza line. This is the minimum amount of defense one needs to protect against the threats most likely to occur based on the evidence and ease of exploit. Since the name “Mendoza” line might be a little misleading: it’s not defined by Mario Mendoza himself but rather by the league’s distribution of players is such that anyone below .200 lifetime is unacceptable. In the same way, regardless of size, a security team is playing against a set of attackers or attacker skill levels. As the team matures, it can ward off bigger and badder threats, defend against more nuanced attacks, detect them earlier. But the reason the Mendoza line is an accurate description of what Infosec teams should aim to achieve is that it is an external metric.

Benchmark against others, not against yourself.

There is a set of necessary but insufficient metrics that any team must incorporate into their day to day to have a chance of being successful. These are the metrics that measure up against something external. The difference here is that you can’t cut your security team – instead you must seek to improve it in order to reach (and surpass) the Mendoza line. In this sense, the Mendoza line is both a measure of minimal safety and of program maturity. The harder question is how does one define said line?

One way is using a free, new feature in RiskDB. “Trending” vulnerabilities are those which are pervasive in our dataset, and hence a good estimator for targets of opportunity. The FAQ will explain how we calculate it, and if you have further questions or suggestions feel free to reach out to me personally.

This way, while a security team might evolve, grow, expand its’ scope, the way in which it benchmarks itself doesn’t change. The theme here is information sharing. We need to find good ways to do so such that security teams can start remediating the truly important issues and benchmarking against external threats, not against their own methodologies.