Scaling Appsec on Netflix (Part 2)

Posted on

By Astha Singhal, Lakshmi Sudheer, Julia Knecht

Netflix’s Application Security teams are responsible for securing the software footprint we create to run the Netflix product, Netflix studio, and business. Our customers are the Netflix product and engineering teams that create these software services and platforms. Netflix cultural values of “uncontrolled context” and “freedom and responsibility” strongly influence how we maintain security at Netflix. Our goal is to manage security risk to Netflix through clear, thoughtful security guidance, and by providing risk context for Netflix engineering teams to make pragmatic risk decisions at scale.

A few years ago we published this blog post about how we organized our team to focus our bandwidth on scalable investments as opposed to just traditional Appsec functions, which didn’t scale well in our rapidly growing environment. We considered the idea of strategic security partnerships and investments in automation to create more leverage for application security. This became the basis of our current organizational structure with teams focused on Appsec Partnerships and Appsec Engineering. In this operating model, we provided critical Appsec operational services to Netflix – including bug bounty, pentesting, PSIRT (Product Security Incident Response), security reviews and security training Developers – via shared on-call rotation.

Team structure v1

In recent years, this model has allowed us to focus on investments such as Secure by default for basic security checks, Security self-service for clear and actionable advice and Large-scale vulnerability scanning for software supply chain security. We wanted to share an update on lessons learned from this model, how our needs have evolved, and where we plan to go from there.

Among the most notable wins, we were able to use this scale-focused approach to produce application security for our rapidly growing studio engineering ecosystem, standardize the security foundation for all enterprise applications, and create paved roads to provide default secure authentication and authorization capabilities for core data engineering tools. We focused on improving overall security assurance rather than just preventing vulnerabilities. We are now extending this approach to other parts of our ecosystem. This mindset has also enabled us to invest our high-end service capability towards reasonable residual risk and standard advice so that we can reduce the need for long-term high-end commitments (e.g., investment in an API proxy that provides basic security checks for free instead of testing all the apps that would end up sitting behind that API proxy). This approach has also allowed us to build strong relationships with Netflix’s core engineering teams (data platform, developer tools, cloud infrastructure, IAM product engineering) who will continue to serve as central leverage points for long-term security.

However, it wasn’t all sunny and rainbow. On the partnership side, the bespoke nature of each partnership means that there is no consistency and redundancy in the operating model and associated partnership artifacts (e.g. security strategy and roadmap, threat, monitoring of deliverables, residual risk criteria, etc.) . This leads to insufficient context sharing and high operational turnover whenever we have personnel changes. The partnership charter has also grown sideways in the infrastructure space as we stack our bets on infrastructure components (like Service Mesh, Container Platform, etc.). The skill sets and depth of domain in these partnerships further diversified the skills of the team. But this is a trade-off on our ability to meet generalized Appsec on-call needs, like triaging bug bounty with high consistency. Since partnerships focus on long-term strategic initiatives, the payoffs can be few and far between and it can be hard on team motivation. We also found various areas where security partnership work impacts the security product solution and it can be difficult to identify appropriate handover points.

Moreover, as the complexity of our ecosystem increases, the goal of a “one PoC in information security” becomes increasingly difficult to maintain. The team is now investing in consistency and scalability partnership artifacts and communication channels, better redundancy and context sharing on the team with more accurate squad operating models engagement criteriaand definition of fact for partnership missions.

Our Appsec engineering team creates products to help us scale, for example: a dynamic asset inventory that understands the nuances of our bespoke engineering ecosystem and how our applications and data relate to each other. This has evolved their identity to become a software engineering team that focuses on security issues as opposed to a security engineering team that writes code/software. Our hiring has reflected this change, and we’ve added more dedicated software engineers (SWEs) to the team to help us develop software. With this change, we integrated engineering best practicesand our products have appropriate investments towards reliability and sustainability. As the team moves towards more software engineering-focused talent, the ramp-up to support Appsec-focused shared custody has been difficult.

Although originally designed to support AppSec use cases around providing self-service developer guidance, the focus on rich data and relationships we have in our tools, in particular our asset inventory, has increased. Therefore, we have continued to invest in making our solutions scalable and accessible, so that security engineers can more easily obtain the data they need to manage security use cases. We also discovered, through interviews with engineers, that self-service advice is not enough on its own. Going forward, the team is investing to better understand our customers’ use cases and shifting our self-service story towards more contextual and insightful automated guidance to ensure developers have everything they need. to make truly informed decisions about the security of their applications (similar to how they might make decisions about resiliency or other products).

As Netflix’s business and engineering workforce has grown, our software footprint has also grown and become more heterogeneous. At the same time, partnerships have become increasingly strategic and engineering has become increasingly software-centric. As our team specialized, what became apparent was a loss of strategic focus for our AppSec Professional Services charter. These services now require more dedicated strategic investments as volume and support needs have increased. So, we are now building dedicated capacity focused on those critical services that are significant investments to make and can no longer be effectively served through a shared on-call Appsec. It will be our “Appsec Reviews and Ratings” function and we are hiring early-career Appsec engineers to join this group.

Team structure v2

We will continue to learn during this next phase of our program’s evolution. We hope to continue to share these learnings with the broader community interested in evolving product and application security.


Scaling Appsec at Netflix (Part 2) originally appeared on Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Leave a Reply

Your email address will not be published.