The detection and response process is doing a really great job scaring the pants off all of us about what’s out there and what the trending threats are. But the real challenge I’m seeing at the core level is around the "people problem". How do we staff to keep up with all this? And the "noise problem." I’ve got a bunch of products and prevention tools, but how do I actually make sense of what’s going on and know where those gaps are? Hopefully we’ll shine some light on that for you guys today. We’re gonna talk through some war stories, some things we’ve seen in customer environments and how we actually helped our customers triage those problems. We'll discuss how we resolved those situations in a very timely fashion, so that these threats, once they get through your parameter, or the tools and securities you have in place, they don’t do as much damage as could potentially be done if they sat there undetected for a long period of time.

I think it’s funny what people think about when they think about a SOC, or a security operations center. It’s generally what we see in the movies. A bunch of screens, some flashing lights, one person behind about 10 screens. I’ve walked through some of these environments and I always ask, you know, do you make sense of this, or is this valuable to you? And they answer No, most of it’s that the screen is directly in front of the person, and it really helps auditors in compliance when they walk through and see that we have that. I think there’s a variance between the people who actually do the work, the hackers, the good-hackers if you will, and what they’re actually doing with their tool sets. But the most important thing about having a SOC is to protect your data. It is why we all have jobs in IT, right?

One reportable breach is extremely damaging to your reputation. And everyone has something that cyber criminals want. I’ve talked to manufacturing companies; I’ve talked to concrete companies. And they’re like yeah, we’ve never had an incident, we’ve never been hacked, don’t have anything people want. I’m like, do you have computers? Yeah, we’ve got like 500 computers. Well, then you have something people want, right? Maybe they’re sitting on your web browsers right now because, you know, the web-based crypto coin. Mining is definitely a very common threat in the landscape, right? The other thing that we see beyond just the trending threats are our customers and the people we talk to are getting a lot of pressure from their vendors. So the whole vendor risk management conversation is becoming very real, very quickly, and I think it’s because the wolf, if you will, is showing its teeth when it comes to compliance. And especially the regulated industries like finance, health care, and manufacturing that may do work for government contracts, or organizations of that nature, right? Those sorts of large companies, and there are a lot of them that work and fuel a lot of the mid-sized companies and small businesses. They’re starting to expect that you have the same security that they can afford, which is just not a reality for most organizations. And so that constantly makes people have to get more done with less. 

In a constantly changing threat landscape, security is a big challenge. And vendors really don’t make that easier all the time either, right? The latest and greatest always costs more money. And security is never-ending, right? The threat landscape changes every year; it changes daily. So if you have your parameter secure, that’s great. You know, every time the refresh cycles come up for firewalls, for anti-virus, it’s always good to take a look at what’s out there and pick your best partner for the reasons that are best for your environment. Doing more of the same and buying point products might not be the best way to really help you.

That layered approached, the defense in depth strategy, the "hey I’m going to beef up my parameter, I’m gonna spend money on that shell," that’s great. When you look at what the big companies are doing they are building security operation centers, they are buying SIM products, they are aggregating all their log data they are adjusting their network flow data, they’re monitoring what their users are doing everyday. And they often have teams of 8, 12, in Target’s case 200 people, monitoring every particular application, anything they can see from an anomaly, and they’re buying tools and threat feeds and anything they can to integrate into those systems to help really narrow down their detection and response time. That’s complicated. I think everyone wants to be there though, right?

But there’s a big gap, right? And that is hard to address because it takes people, it takes time and it takes resources. And so that’s really where Arctic Wolf aims to play and  help. I also hear the AI buzz word a lot. I think automation is what’s been talked about a lot more. And both are great. The reality around AI though, is if it was really that great then a WatchGuard would buy it and integrate it into their system, or so would a CISCO, or so would Palo Alto, and then it would be a next generation firewall. It becomes another parameter tool at that point.

People create new things everyday because they’re creative people often that are doing those types of attacks. Prevention’s not enough; you heard about all that’s out there so I don’t have to go into it. What I do wanna talk about is here in Minneapolis, we have a big gap. More of a talent gap, a "hiring costs a lot of money" gap. This is from Glassdoor. You look at Indeed or any of the other websites are out there and that’s the truth. Hiring someone with even a few years of experience in security that have built a security operation center or done this detection is not easy or inexpensive. To build it out and find someone with experience, let alone talking about the tools and everything else that go along with it is a huge challenge. Big businesses such as Target, Best Buy, HealthPartners, Optum really drive up the cost of security professionals as well, which makes the challenge even tougher.

And then you could try, you know, growing a unicorn. I hear Evan at FRSecure talk about that a lot. They’ve done a really good job doing that. If you haven’t checked out their CISSP Program for growing unicorns, it’s really cool and I highly advise taking a look at that. But the point I’m trying to make is, you’re not alone.

All companies have those same challenges. There’s a lot of noise out there, and there’s a lot of security products out there.

Every trade show I go to I’m just amazed at the three, four, five, sometimes even ten vendors that I’ve never heard of before that go to these big shows and are trying to stamp their presence in the market. And the do-it-yourself kind of detection response approach is complex, it’s hard to do because it takes people, processes, technology, time, expertise and experience to make it work effectively.

And so that really leaves people kind of scratching their heads saying, "well, how do we increase our security posture?"

Should we implement a SOC; do we outsource? If we can't hire more people, do I just keep doing more of the same and buy more products? Those are often the questions I see people ask, and so what I want to talk about next, and Bryan’s going to take over here and kind of give you a view from a technical standpoint into what a SOC looks like, what’s actually going on behind the scenes. Because it’s hard if you’re trying to do it on your own, and it’s extremely important to get value out of outsourcing detection and response. Understanding what’s going on, what the context is, all the forensic details correlating across everything is a lot to take in, process and implement.

Most companies spend a large amount of time looking at the signature of things, learning algorithms and running  detection tools, trying to look for patterns, changes in behavior, signs of something new that might be in the environment. 

If you think about how much machine data is in an environment, how many log sources we can pull in and getting that all in one place for a security professional to review should really be the goal of a SOC.

There is a ton of noise, so reducing the number of false positives certainly helps. This makes humans more efficient, gives them more time to do other things, like threat hunting, and really helps cut down on wasted time within the SOC for those resources that are expensive and hard to find. 

So we're trying to find the signals through the noise, and doing it quickly and efficiently is really key. Being able to do forensics, using tools like VirusTotal, and quickly analyzing if something is bad, or something new that needs an alert created. 

We're always looking at kind of an evolution of security as your parameter is built out. So pulling the log data in is really important. Looking at, kind of that as a whole picture here, we pull in some other things like internet of things devices, webcams, printers, all of those types of things. Looking at it from a narrow-based approach, as well as at log data, allows you to capture all that. Conversely, if you had a dedicated end point as your primary detection, you’d have gaps around devices that aren’t managed because you probably can’t install an end point agent on a webcam or a printer, a thermostat or those types of things.

Doing all that hard work in real time correlation across is really important for a SOC to do. All the machine learning and things that that make people more efficient are important, but that human element still has to be there. They’re not going to be removed from the security equation anytime soon. And they’re also going to be the ones helping with the forensics and threat hunting and things like that, and continuing to grow the threat feeds and look for those new anomalies that emerge.

The workflow behind the scenes in a security operation center involves data, making sense of the noise, or alerts that are created out of that. Many IT departments are understaffed; you’re working on infrastructure projects, there’s other business priorities going on. When I’ve asked companies and people, how are you handling or triaging alerts today, the common response I get to that question is, "oh I’ve got an email folder set up in my inbox. All of our systems feed alerts into there. We get around to ‘em, ya know, we try to comb through ‘em every morning before we’re done with coffee and address what’s needed."

That’s just too much for any single person, or even team of people to do, right? So if you don’t have automation built in, you don’t have ways to trim down that noise, all you’re doing is wasting time looking at things that may not be important or relevant. We see far too often you’re getting alerts from three different tools in your environment that are all throwing false positives, but you’re still going and checking ‘em, right? And so, the point I wanted to make here is the step from taking an actual alert and turning it into an incident.

It’s hard to conceptualize unless you’ve done that kind of grunt work before, and it’s also not easy to conceptualize if you haven’t been there on the ground level reviewing all that stuff. Because reviewing logs is probably one of the least glorified jobs out there when it comes to security work. It’s monotonous, it’s tricky, it takes a trained eye, and ya gotta know about everything you’re working with too. So the point I wanted to make there is kind of, that funnel, if you will, of data and taking the large observation number and turning it down to the meaningful stuff that actually you pay attention to is that hard work that goes on.

Right, and a lot of these aren’t uncommon things in IT. Quick detection and reducing the dwell time is really what makes a breach less severe. It’s normally not a single point product that would be your only clue that something bad is happening in your environment.

One of our examples is a healthcare provider. Malware was delivered through a phishing email, which we know is pretty common. We started seeing malicious web traffic and things like that and started helping, narrow that dwell time and the detection capability. We worked with the customer to block subsequent access to that website, so we cut it off from spreading, and then quickly identified who had been impacted so that the machines could be re-imaged with a valid backup and things like that. It really helped that instance from becoming something more of a breach or something that required disclosure.

- [Ben] Phishing, and you heard it earlier, is probably one of the primary ways that attackers get into environments because it’s often our people letting ‘em in. This particular story actually came from a phishing incident we helped a client of ours who’s a luxury cruise line. They had an email come out from their CEO, which was a spear phishing attack that spoofed the CEO, and it was sent to the whole company. And we started seeing things happening and so we quickly elevated the incident to the customer, and told them what we had seen. Fortunately in this instance a lot of the attack had been launched over standard HTTP, so we were able to actually see what the users had put into web forms, or which people had input data there, and helped them reset user passwords and address each particular individual user, suggesting that they both change their passwords at home, at work, force the password resets at work. You know, anyone that we saw any sort of malicious activity or downloads on we suggested they re-image the computers. We were able to help them troubleshoot this email that went out to the entire company list where a small percentage fell for this spoof and actually input some pretty serious information. We were able to see that issue and immediately solve it so it didn’t get left for a week or two with company passwords floating around out there, and access to launch the attack even further.

Three years ago a customer of ours on the west coast saw us at a trade show, was actually an IT manager, and brought us into the company. They said they need our services really badly and that they had lacked in this area for awhile. They pitched us to their leadership, got a contract in place, and a few weeks into monitoring the environment we started seeing some pretty strange activity in the evening hours. It became very evident to us that he was the only one logging into servers at particular hours of the night, when the servers weren’t being used by the company. He was using them for his own pretty intricate web he'd spun together doing Bitcoin mining three years ago before the big boom. And we saw he was doing this, quite effectively, and had been for a period of time, when we started digging back through the logs we had access to. And so we actually had to elevate that to his leadership and he ended up being terminated as a result of the situation. And he was the person that brought us into the company.

There’s a million different ways insider threat can bite a company, whether through data exfiltration or malicious intent, whatever it may be, that’s another form of attack. That’s not often someone externally doing that, but still you want to be aware of it, be able to dig into forensically. And without looking at the logs, with looking at the context of everything that’s going on in your environment, sometimes that can be hard.

Another example we have shows that often security is against convenience. What is secure isn’t convenient. And here was a hospital that, basically doing things the right way was too difficult so it was easier just to make life easy, run an FTP server exposed to the internet, allow anonymous credentials, and just fail to upload patient data there. Someone on the internet obviously thought that was really convenient as well, decided to load Malware that way, and then to do things a little differently ended up using DNS as kind of a covert channel. We would’ve detected that earlier with scanning and things like that, just FTP exposed. The internet overall is a very bad idea, but then taking us a step further looking at kind of DNS traffic, as well as communication and things like that would really help shore that up. So we help detect on that early before a large number of records were disclosed in the landscape, kind of knowing what was exposed helps lessen your fines or your requirements. 

Another example is a war story I call "moving the goal line." It was really interesting to me hearing this pain from a customer of ours and what they were going through. It was mostly stemming from a vendor risk management conversation.

This was a Midwest-based law firm that was doing business with Bank of America. And three years ago Bank of America was saying, hey, everyone we do business with has to do external scanning quarterly and then internal scanning pen test once a year. This was three years ago. A year later in 2016, they upped that to quarterly. Internal scans monthly, external scanning, that’s a requirement. And then in 2017, the client was asked to do weekly external scans, monthly internal scans. And they had a three person IT team. That’s more than a full time job for one person to keep up with, right? And so, fortunately for them we were actually partners with them. External scanning is something we do as incorporated in our service monthly as part of what you would get out of a security operation center.

So we were able to make some tweaks for that customer and accommodate their demands by the client and really help them. Moving the goal line had a very big impact; it had them really stressed out. It was causing them a lot of pain and suffering. By outsourcing and driving down the cost, they were able to accommodate our services and repurpose their resources elsewhere, which was really helpful to them.

Another big company, Medtronic, was requiring that they proved vendors that they worked with, or clients that they worked with, were doing 24 by seven active network monitoring and log management or log aggregation. And since this client couldn’t do it and were kinda under a time constraint they actually ended up losing, or not being considered for that piece of business, right?

We’ve seen a lot of changes from how the Department of Defense, the government, big companies, etc., that require a company they do business with to have to abide by over 100 different security controls that they didn’t have to abide by a week before to even maintain the contract with the Federal Government. That’s alarming, but it’s reality.

Don’t give up or stop because you think you can't afford a security operation center. What I want you all to know is there are options now.