Smashing The Shift Left Unicorn

Every few years, a new cybersecurity fad emerges promising to revolutionize the way we do business. Once upon a time, we heard of DevSecOps. Now, the new prom queen is Shift Left, and a market of vendors — old and new — want to sell you their wears so you can be as cool as the stock photos in their marketing.

I respect what Shift Left looks to accomplish because this approach attempts to meaningfully integrate security as a core component of the software development lifecycle. Yet, my experiences as a CISO prevent me from pushing all of my chips all in on this fad. My time on target has taught me that technology and tools cannot solve a human-based problem. To realize the promise of the Shift Left unicorn, security must become an integrated and habitual part of all software development through behavioral and cultural shifts that prioritize security throughout the software development lifecycle.

You cannot buy security and privacy off the rack. Rather, leaders need to take a hard look at the specific challenges they face and make changes to address them, in their context.

Opening Shots

As a CISO, I take security very seriously, it’s literally my job. As an executive, I am accountable to both our business and our users. This means that I must prioritize both corporate security and product security, which require separate strategies as they are different domains.

For some time, I did not understand the difference between these two very distinct areas. It led me to make mistakes — some that are hard to come back from. If you have not gotten into the guts of software development, product management, and DevOps you may not realize all of the classic mistakes that your security organization is making.

The purpose of this article is to share some of the lessons I’ve learned to support security teams and engineering leaders so they can successfully and confidently build safe and trustworthy products and organizations.

False Objective

If you speak with the seemingly endless army of Shift Left evangelists, they’ll happily tell you that their strategy will reduce the number of production vulnerabilities. That is fair. But it’s also a totally false objective.

Limiting production vulnerabilities is what security people care about but that’s only part of the story. It’s an output — not an outcome.

The objective of an appropriate Shift Left strategy should be to reduce the number of security work items generated altogether. Merely shifting responsibility for who detects and manages vulnerabilities during development will not build stronger developers, increase flow, reduce rework, or delight your customers.

The Shift Left Precursor

As a believer in not reinventing the wheel, I’ll turn to Wikipedia for quick TL;DR on the scores of works published to discuss this new phenomenon.

Shift-left testing is an approach to software testing and system testing in which testing is performed earlier in the lifecycle. It is the first half of the maxim “Test early and often.” It was coined by Larry Smith in 2001.

The term itself has nothing innately to do with cybersecurity. Test-driven development (TDD), which is the precursor to this current flavor, has been around for nearly two decades in the quality assurance space and has proved incredible results. TDD enables a few key outcomes:

1. Find bugs early and often
2. Better designed, cleaner, and more extensible code
3. Code that meets requirements and acceptance criteria
4. Better code refactoring processes
5. Happy developers, happy teams

None of this is out of market with the Shift Left movement in cybersecurity. Yet, if that’s true, then how is it possible that our new security mantra will be anything less than fantastically successful?

Go To The Mountain

Security teams are all too often somewhere between out-of-sync or at odds with software development and product management teams. That isn’t because software developers and product managers do not care about security. Both teams know that they cannot ship market-ready products and fulfill expectations if what they publish cannot be trusted. In most organizations, the problem is not their willingness to act but rather security’s all too common inability to align with the organization’s processes, incentives, and workflow.

Countless tomes have been written on the topic of DevOps. Three top contenders are The Phoenix Project, Accelerate, and The DevOps Handbook. If you work in cybersecurity and have not read and internalized the learnings in these books, you will not be successful in modern organizations. While these works do a truly sad job discussing how security fits into the broader organizational picture, they do an excellent job teaching what is important to software developers and product management organizations.

Dear Security Organizations,

Remember the famous words of Sir Francis Bacon: If the mountain will not come to Muhammad, then Muhammad must go to the mountain.

Regards,

CISO Of The Future

Create Alignment

Developers are often measured by the number of features released that meet the requirements and acceptance criteria. They are often not measured on the security and trustworthiness of that code. If security organizations want this to change, they need to fundamentally rearchitect their approach to ensure that organizational priorities include not only the quality of code and the number of features released but also to ensure that the code meets an established set of security and privacy standards and requirements. This isn’t rocket science: it’s your culture.

Every time I talk with security people, I hear them go on and on about how their developers just don’t pay attention, that security teams are rarely consulted with, and something like “kids these days just don’t understand.” On balance, I reject that. I tend to think that security people are often to blame for security problems. It’s time for security to get on the same team with development and product — development is, in the end, the culmination of the work of the broader organizational team, not teams.

Here are a few key questions that developers and product managers have asked me over the years:

1. Where are your requirements documented and how are they shared with developers?
2. Which security frameworks does the security team want us to use, why?
3. On what board can I find the Security User Stories and Evil User Stories for this sprint?
4. Where can I find the library of security dependencies for this build?

To be clear, I do not have all the answers and am myself just coming up to speed on some of this. But I’ve learned that being of service to our developers helps them solve problems before they become consequences for our business.

It’s The Workflow, Stupid

Recently, I started talking with some friends in Product Management at a Fortune 500 company about their own security organization and what they thought about it. I was floored by their responses.

We’re three quarters behind our release schedule because our security team does all their work at the end of the build.

They just email my dev managers a PDF report of everything we did wrong and expect that to help. It doesn’t.

I don’t even get tickets from these people — just emails and angry phone calls.

They’re pissed at my devs but I’m pissed at them — they don’t seem to know how software development works.

The reality is that there is some good coming out of the current fascination with Shift Left in cybersecurity. It’s good that developers get their hot little hands-on tools and platforms to do security testing closer to the beginning of a build than when they’re ready to promote to production. But don’t be fooled — tools alone will not save you.

Security teams need to deeply understand the workflow of developers before going out to buy kit. They must clearly align their expectations with the reality of the software development lifecycle and integrate security tools into Continuous Integration and Continuous Deployment (CI/CD) pipelines. But more than this, they need to understand that tools without understanding are the security equivalent of lipstick on a pig.

Piles of Tickets

Security teams that launch Shift Left strategies know that the realities of the experience do not match the marketing they signed up for. This is true for a few reasons:

1. Developers often ignore the IDE-based alerts, find them distracting, and disable them.
2. Security tools still generate the same number of tickets no matter who manages them if developers make mistakes.
3. Developers may not know how to solve the problems the security tools find because they are software developers — not security engineers.

The resultative effect of buying Shift Left tools and strategies is that some value is extracted but not nearly the kind of return on investment than you might think. The impacts, however, are felt by the whole of the business:

1. Now we have piles and piles of tickets, which force organizations to either reprioritize work — kicking feature work out of the sprint — or pushing security work items to a later date, which often results in delightful fights between Scrum Masters, Product Managers, and security engineers.
2. Everyone freaks out because now that the security information is visible, a ton of people think you’re going to get hacked. Trust me, this is not what security teams want. While they might get priority, they will also get the wrath of the CTO for not having been more effective in the first place.
3. Developers start to feel defeated and architects begin to feel frustrated. And who wouldn’t?!? All of your test-driven development for Quality Assurance passed testing and now someone new is telling you that your work doesn’t smell like a bed of roses.

Shift Left Is Incomplete

When platform companies come to me and extoll the virtues of their product, I take a healthy swig of pessimism juice from my Hydroflask. I know what other seasoned security people know: products are not solutions.

Any Shift Left strategy needs to also focus on what is not in the product.

Remember our key objective: reduce the number of work items generated altogether. To achieve this goal, you’ll need three tools in addition to a great security testing tool suite: developer education, patterns and libraries, and a change of workflow.

Invest In Your People

Developers will need training and education. Most organizations underinvest in their workforce, which is a shame yet, there are great security training products purpose-built for developers.

What befuddles me is this quandary: If developers are the point of origination for vulnerabilities and misconfigurations, why are so many organizations willing to invest $50,000 in a security testing suite of Shift Left tools but not prioritizing proportional investment in developer training and education so that they can fully realize the promise of Shift Left strategies?

Answer: Security teams are confusing outputs with outcomes. They often focus on meeting compliance requirements, which mandate security testing tools. As a result, security teams do not invest in leveling up developer security education because their priorities are misaligned with the goals of the business. Until security teams start being judged on how they impact release velocity, change will come too slowly. But if they truly want to increase the level of security in their organization, they need to focus on the level of education and capacity in their software development teams.

Pro Tip: Set built-in or integrated security training for developers as part of your procurment success criteria.

Support Your Architects

This is an area where I have previously dropped the ball — but I’m committed to picking it back up again and I suggest that you do too.

While so much of any platform’s code is written by developers, many patterns and dependencies are reused. Security organizations should partner with software architects to identify the libraries/dependencies/patterns in wide use and work together to ensure that these tools meet security requirements. This should be an exercise in collaboration, which is at the very essence of Shift Left.

Pro Tip: If you’re stuck here, work with your engineering management to identify a great contract resource to create outcomes for you in your stack and then assign those software resources to an owner in your architecutre team to manage and improve.

Change Your Processes

Shift Left tools have their limits. At some point, security teams are going to have to get dirty and use their own tools like Metasploit and others. The question isn’t if that should happen but rather when it must happen.

Timing is the essence of life, and definitely of comedy. — Bob Hope

Security is only different from comedy insofar as the people who work in it are not funny. But the lesson from Bob Hope still stands.

What kills me about security people is that they think they’re special. While they have a special set of skills, they are not Liam Neeson. When security teams need to run a pen test, they should do it in coordination with their Quality Assurance team. This is not to say that security and QA are one and the same — they are not. Rather, it is to say that there is an appropriate time and place for security testing — when the other testing team is doing their work. While you’re searching for Cross Site Request Forgery, QA is looking for Syntactic Errors. This will help your organization increase flow, reduce rework, and ship safer code faster.

Pro Tip: Be good to your QA manager. That person has more on their plate than you know. You’re both in the same fight — just on different teams.

Closing Shots

In case you are wondering, no I am not a hater of companies touting their Shift Left wears. I am actually a fan. But products are not strategies and a lot of people cannot seem to tell the difference. Until that changes, we will not move the needle against attackers and we will not help our organizations create the business outcomes we all want.

Be pragmatic, think beyond the tool, and work across your organization. It's team — not teams.