The Evolution & Future of DevOps
Highlights from an awesome research report by Contrary Research
I recently came across a super interesting X-thread (still feels weird to write X instead of Twitter…) from Contrarian Research about the evolution of DevOps. Check out the thread here and the full report here. In the following, I summarize some of the key learnings and complement some additional sources and insights.
What is DevOps again?!
“DevOps can be best explained as people working together to conceive, build and deliver secure software at top speed. DevOps practices enable software development (dev) and operations (ops) teams to accelerate delivery through automation, collaboration, fast feedback, and iterative improvement.” (source: Gitlab)
Security teams play a key role to ensure that software is delivered safely. Engineering teams plan, create, and verify code. Operation teams ensure successful release, packaging, configuration, and monitoring. DevOps practices allow to shorten the software development life cycle, sometimes from months to hours.
Core principles:
Automation of the software development lifecycle. This includes automating testing, builds, releases, the provisioning of development environments, and other manual tasks that can slow down or introduce human error into the software delivery process.
Collaboration and communication. A good DevOps team has automation, but a great DevOps team also has effective collaboration and communication.
Continuous improvement and minimization of waste. From automating repetitive tasks to watching performance metrics for ways to reduce release times or mean-time-to-recovery, high performing DevOps teams are regularly looking for areas that could be improved.
The DevOps lifecycle consists of eight phases representing the processes, capabilities, and tools needed for development (on the left side of the loop) and operations (on the right side of the loop). Throughout each phase, teams collaborate and communicate to maintain alignment, velocity, and quality. (source: Atlassian)
To make DevOps work well, you need a team culture where everyone works together smoothly. Pipelines, code merges, testing, validation, and a high degree of automation are required to get your code from the drawing board to being used without problems. This is where Continuous Integration (CI) and Continuous Delivery (CD) come in. CI means automatically testing your code as you write it, and CD means deploying your code very frequently. Using this continuous and iterative process to build, test, and deploy can help avoid bugs and code failures.
DevOps Tool Stack
DevOps teams usually put together a customized toolchain to connect the various people and workflows that consist of open-source and vendor tools. That leads to a very fragmented vendor landscape.
Most DevOps vendors specialize in a particular area of the toolchain, such as planning, issue tracking, source code control, code writing, testing, configuration management, deployment, or monitoring.
Choosing the right DevOps tools for an organization can be a complicated process due to the involvement of various stakeholders. Executives aim to maintain system availability and manage costs, whereas developers prioritize performance and developer-experience.
DevOps Drivers: Cloud + Open Source + ML (???)
Undoubtedly, in the last decade, Cloud computing and open source have been major factors in propelling the adoption of DevOps practices and tooling.
Cloud computing and the continued abstraction of different software components (containers, serverless, APIs etc) have increased the rate of production for software.
“Today you can host your application on AWS, store your data on Snowflake, serve up content through Cloudflare, build out your client side application with Vercel, monitor your applications with Datadog, and integrate directly with thousands of other applications like Twilio, Stripe, and Okta via API.” (source: Contrary Research)
This brought down software development cycles from years to months/weeks.
Additionally, with the rise of Open Source Software, developers were given the power to use the very best tools for any aspect of their project. This democratization increased development velocity and the quantity of produced code even more.
But, all this resulting complexity has to be managed.
This is where DevOps enters the stage and shines.
It can certainly be argued that in the age of LLMs and AI-generated code, the amount of code and complexity will increase even more, making sound DevOps practices more important than ever.
The DevOps Opportunity
The DevOps software market is estimated to be ~8bn in 2022 but will grow at a whopping 20% CAGR to ~70bn by 2032.
According to Bain, only 50% of companies have implemented DevOps practices in their organization, and only one in five companies say they have the right tools and automation capabilities to scale DevOps.
This indicates that the DevOps revolution is still in its infancy.
The Future of DevOps
Some predictions according to Contrary Research and Gleecus:
Consolidated Toolchain
As it becomes more critical to drive efficiency in the software development lifecycle, companies will look to move more of their DevOps stack onto the same platform. This will favor companies that have built multi-product platforms over best-in-class point solutions, and set the stage for what companies will become dominant in DevOps going forward —> anticipating the “you can’t get fired for buying XYZ”-moment in DevOps
Greater Shift Toward Serverless Computing & Microservices Architectures
DevOps is shifting gears. Expect serverless and microservices to become the driving forces for agility and resilience. By managing the distributed systems and infrastructure burden, these technologies liberate engineers to focus on building scalable, efficient applications on demand.
As stated in the “The State of Serverless” report by Datadog, major cloud providers continue to see significant serverless adoption.
Serverless computing platforms, including AWS Lambda, Google Cloud Functions, and Azure Functions, offer compelling value propositions for DevOps teams. Their ability to automatically manage and scale cloud infrastructure enables significant improvements in resource efficiency, cost optimization, and operational agility.
Microservices also drive the push to a new era of agile innovation in DevOps. By deconstructing monolithic applications into smaller, independent services, microservices empower parallel development, streamlined processes, and faster delivery cycles. Teams work in parallel and this significantly reduces the time it takes to bring products to market.
Emergence of NoOps
NoOps (short for "No Operations") is a concept in the DevOps world that envisions an IT environment so automated and abstracted from the underlying infrastructure that there is no need for a dedicated team to manage software in-house. The idea is to reach a stage where software can be deployed, managed, and monitored without the traditional operational overhead. Powered by AI, Infrastructure as Code, and serverless computing, NoOps transforms systems into autonomous entities. Algorithms anticipate and manage resources, security patches itself, and rollbacks occur before issues even flicker.
DevOps as a Service (DaaS)
There's a rising trend where companies are increasingly turning to DevOps as a Service (DaaS) to lighten their operational load. This method involves outsourcing DevOps roles to external service providers. Adopting this model, businesses can incorporate DevOps methodologies to create stable and repeatable workflows, but avoiding the costs and complexities of in-house DevOps management.
Moreover, DaaS empowers organizations to leverage the expertise of seasoned DevOps consultants and professionals, accessing cutting-edge automation tools. This not only saves time and resources but also ensures a swifter return on investment (ROI). The prevalence of DaaS is anticipated to grow significantly in 2024 and potentially continue as a prominent choice for years to come.
Increasing relevance of DevSecOps
With the pace of software development accelerating, there's less opportunity for conducting security reviews after development, before the software is released. Integrating security checks early in the development process, known as "shift-left," is becoming essential. For any company aspiring to be a significant player in the DevOps arena, making security an integral part of the development lifecycle is increasingly becoming a fundamental requirement. Shift-left, along with the increased emphasis on involving developers in the security process, makes the security review process more proactive.
Before vs After “shift-left”: more secure, faster time to market, more intertwined processes and teams
According to Bain, three out of four companies plan to incorporate security into their technology stack to develop fully integrated DevSecOps.
Infusion of AI/ML
The convergence of MLOps with DevOps is becoming more evident, due to the parallel nature of the workflows by software engineers and data engineers. This integration broadens the Total Addressable Market (TAM) for DevOps platforms by including those personas. Apart from expanding the user base, AI is increasingly becoming an integral component of DevOps products. The extensive data accessible to these DevOps platforms also presents opportunities to further automate various stages of the development lifecycle.
Also, using machine learning in DevOps streamlines the software delivery process. Intelligent algorithms can automate various stages, including the building, testing, and deployment of applications. Within the scanning and testing aspect of DevOps, companies like Harness, Snyk, and GitLab use similar models to analyze code and help engineers detect security anomalies, and potential code vulnerabilities using pre-built models that have been trained on publicly available open-system data. For example, Snyk acquired Deepcode to enable automated code reviews in addition to suggesting changes to developers looking for security vulnerabilities.
Another example is Replit, a collaborative browser-based Integrated Development Environment (IDE) that allows developers to code in 50+ languages. The platform is easy to deploy with multiple integrations to make it easy for developers to create their programming environments. Recently, they announced Ghostwriter. Most of that product is built off of open source LLMs that have been optimized for ease-of-use. Ghostwriter offers code completion suggestions in both snippets, and whole functions. There are numerous examples of similar companies and ML approaches in the DevOps tool stack.
At First Momentum, we recently invested in a company that is following this market hypothesis in the ML-driven Software QA segment. Octomind is leveraging AI-agents to autonomously test and de-bug user interfaces, fully integrated in the CI/CD pipelines and workflows.
Check out their product here: https://www.octomind.dev/
Hi David,
DevOps is definitely here to stay! Empowering developers with tools and processes to bring their new features into production faster and more secure.
The platform all these tools and companies build on top of is currently Kubernetes. How do you see the role of Kubernetes for the future of the DevOps ecosystem. As you showcased the serverless architecture, it would interested to compare the benefits and drawbacks of it compared to a platform built on top of Kubernetes with the right tooling.
Best, Philip