Skip to main content

Comprehensive DevOps and Modern Software Delivery Toolchain Overview

· 15 min read
Anand Raja
Anand Raja
Senior Software Engineer

These tools cover infrastructure automation (Terraform), CI/CD (Jenkins, Spinnaker), workflow orchestration (Airflow), operations (Rundeck), code quality and security (SonarQube, Veracode), feature management (Split.io), service discovery (Eureka), API management (Kong, Zuul), web serving and proxying (Nginx, Reverse Proxy, Apache HTTP Server), content delivery and security (Akamai, Cloudflare), artifact management (JFrog), analytics (Pendo, Snowplow), Java builds (Gradle, Maven), Java app hosting (WildFly), and Ruby app servers (Puma, Unicorn). Each plays a key role in modern DevOps, analytics, and software delivery pipelines.

Terraform​

  • What: Terraform is an open-source Infrastructure as Code (IaC) tool developed by HashiCorp. It allows you to define, provision, and manage cloud infrastructure and resources using declarative configuration files written in HCL (HashiCorp Configuration Language).
  • Usage: Infrastructure as Code (IaC) for provisioning and managing cloud/on-prem resources.
  • Key Features:
    • Provisioning Infrastructure: Create, update, or delete resources (servers, databases, networks, etc.) on cloud providers like AWS, Azure, GCP.
    • Infrastructure as Code: Define and manage infrastructure using code stored in Git.
    • Multi-Cloud: Supports AWS, Azure, GCP, and more.
    • Dependency Management: Handles resource order automatically.
    • Execution Plan: Previews changes before applying.
    • State Tracking: Remembers current infrastructure state.
    • Modular: Allows reusable code modules.
    • Automation: Automates resource management.
    • Extensible: Supports plugins and community providers.
  • Alternatives: AWS CloudFormation, Pulumi, Ansible

How to Implement Blue-Green Deployment in Terraform​

Blue-Green Deployment in Terraform refers to a deployment strategy where you have two identical environments: Blue (current/production) and Green (new version). You deploy changes to the Green environment, test it, and then switch traffic from Blue to Green, minimizing downtime and risk.

1. Duplicate Infrastructure​

Define two sets of resources (e.g., blue and green) in your Terraform code, often using workspaces, modules, or naming conventions.

resource "aws_instance" "blue" {
  count = var.active_environment == "blue" ? 1 : 0
  # ... other config
}

resource "aws_instance" "green" {
  count = var.active_environment == "green" ? 1 : 0
  # ... other config
}

2. Use Variables to Control Active Environment​

Switch between environments using a variable:

variable "active_environment" {
  description = "Which environment is active: blue or green"
  type        = string
  default     = "blue"
}

3. Update Routing or Load Balancer​

Point your load balancer or DNS to the active environment:

resource "aws_lb_listener_rule" "active" {
  listener_arn = aws_lb_listener.front_end.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = var.active_environment == "blue" ? aws_lb_target_group.blue.arn : aws_lb_target_group.green.arn
  }
}

4. Deployment Steps​

  1. Deploy new changes to the Green environment.
  2. Test the Green environment.
  3. Switch the active_environment variable to "green" and apply Terraform to update the load balancer/DNS.
  4. Optionally, destroy or keep the Blue environment for rollback.

Jenkins​

  • What: Open-source automation server for CI/CD. Jenkins can be used for both Continuous Integration (CI) and Continuous Delivery/Deployment (CD).
  • Usage: Automate building, testing, and deploying software (CI/CD).
  • Key Features:
    • Pipeline as code (Jenkinsfile)
    • Large plugin ecosystem
    • Distributed builds
    • Integration with many tools
  • Alternatives: GitLab CI/CD, CircleCI, Travis CI

Spinnaker​

  • What: Open-source continuous delivery platform (by Netflix). Spinnaker is primarily designed for Continuous Delivery (CD), not for Continuous Integration (CI).
  • Usage: Continuous delivery platform for multi-cloud deployments.
  • Key Features:
    • Advanced deployment strategies (canary, blue/green)
    • Rollbacks and monitoring
    • Integrates with Kubernetes, AWS, GCP
  • Alternatives: Argo CD, Flux, GoCD

Jenkins with Spinnaker​

You can use Jenkins and Spinnaker together by letting Jenkins handle Continuous Integration (CI) (building, testing, packaging your code) and Spinnaker handle Continuous Delivery (CD) (deploying your built artifacts to staging/production).

How to Use Jenkins and Spinnaker Together​

  1. Jenkins for CI

    • Jenkins builds your application, runs tests, and creates deployable artifacts (e.g., Docker images, JARs).
    • After a successful build, Jenkins pushes the artifact to an artifact repository (e.g., JFrog Artifactory, Docker Hub).

  2. Trigger Spinnaker Pipeline

    • Jenkins notifies Spinnaker (via webhook, REST API, or artifact registry trigger) that a new artifact is available.
    • Spinnaker pipeline is triggered automatically or manually.

  3. Spinnaker for CD

    • Spinnaker fetches the new artifact.
    • Spinnaker deploys the artifact to your target environment (Kubernetes, AWS, GCP, etc.).
    • Spinnaker can perform advanced deployment strategies (canary, blue/green, rolling updates).

Typical Workflow​

Integration Methods​

  • Webhook: Jenkins sends a webhook to Spinnaker after a successful build.
  • Artifact Registry Trigger: Spinnaker polls the artifact repository for new versions.
  • Spinnaker Jenkins Stage: Spinnaker can trigger Jenkins jobs as part of its pipeline.

Summary:

  • Use Jenkins for building and testing.
  • Use Spinnaker for deploying and managing releases.
  • Connect them via webhooks, artifact triggers, or pipeline stages for a complete CI/CD workflow.

Apache Airflow (and DAG)​

Apache Airflow is not typically used for CI/CD in the traditional sense. Instead, it is a workflow orchestration tool designed for scheduling and managing complex data pipelines, ETL jobs(Extract, Transform, Load), and batch processing tasks.

  • What: Open-source workflow orchestration tool.
  • Usage: Workflow orchestration and scheduling for data pipelines.
  • Key Features:
    • DAGs (Directed Acyclic Graphs) for task dependencies
    • Python-based workflows
    • Scheduling, retries, monitoring UI
  • Alternatives: Luigi, Prefect, Dagster

Rundeck​

Rundeck is used for automating and managing IT operations tasks, such as running scripts, executing jobs, and orchestrating workflows across servers and environments.

  • What: Open-source operations automation platform.
  • Usage: Operations automation, job scheduling, and workflow management.
  • Key Features:
    • Web UI and API
    • Role-based access control (RBAC)
    • Job orchestration and scheduling
    • Integrations with Ansible, Chef, Puppet
  • Alternatives: StackStorm, Ansible Tower, SaltStack

SonarQube (Sonar)​

  • What: Open-source code quality and security analysis tool. SonarQube measures code coverage by integrating with your test suites and showing what percentage of your code is covered by automated tests.
  • Usage: Code quality and security analysis.
  • Key Features:
    • Static code analysis for bugs, vulnerabilities, code smells
    • Supports many languages
    • Quality gates and dashboards
    • CI/CD integration
  • SonarQube can detect some types of vulnerabilities in your code.
    • It performs static code analysis to find security issues, bugs, and code smells.
    • SonarQube identifies common vulnerabilities such as SQL injection, XSS, hardcoded credentials, and more, depending on the language and rules enabled.
    • However, SonarQube’s vulnerability detection is limited to what can be found through static analysis of source code.
    • For deeper or runtime security testing (like binary analysis, third-party library scanning, or dynamic testing), you should use dedicated security tools such as Veracode, Checkmarx, or Fortify.
  • Alternatives: CodeClimate, Coverity, Veracode

Veracode​

  • What: SaaS application security platform. Veracode focuses on security scanning and does not provide code coverage metrics.
  • Usage: Performs static, dynamic, and software composition analysis for security and compliance.
  • Key Features:
    • Scans code for security vulnerabilities (SAST, DAST, SCA)
    • Works in the cloud (no local setup needed)
    • Supports many programming languages
    • Checks both source code and third-party libraries
    • Gives clear guidance on how to fix issues
    • Integrates with CI/CD tools and developer workflows
    • Provides compliance and policy reports
    • Central dashboard to track security across projects
  • Alternatives: Checkmarx, Fortify, SonarQube

SonarQube with Veracode​

You can use SonarQube and Veracode together to improve both code quality and security in your development workflow. Here’s how:

How to Use SonarQube with Veracode​

  1. Integrate Both Tools in Your CI/CD Pipeline

    • SonarQube: Run SonarQube analysis during your build process to check for code quality issues, bugs, and code smells.
    • Veracode: Run Veracode scans (SAST, SCA, or DAST) after the build to check for security vulnerabilities.

  2. Typical Workflow

    1. Developer pushes code to repository.
    2. CI/CD pipeline triggers:
      • Step 1: SonarQube analyzes the code for quality and basic security issues.
      • Step 2: If SonarQube passes, Veracode scans the built artifact for deeper security vulnerabilities.
    3. Results from both tools are reviewed before merging or deploying.

  3. Integration Tips

    • Use plugins or CLI tools for both SonarQube and Veracode in your CI/CD system (e.g., Jenkins, GitLab CI).
    • Set quality gates in SonarQube and security policies in Veracode to fail builds if issues are found.
    • Review reports from both tools and address issues before release.

  4. No Direct Integration

    • SonarQube and Veracode do not directly integrate with each other, but both can be part of the same automated pipeline.

Split.io​

  • What: Split.io is a feature flag and experimentation platform used to manage feature releases and run experiments (A/B tests) in your applications.

  • Usage: Feature flag management and experimentation.

  • Key Features:

    • Feature Flagging:
      Enable or disable features in your application without deploying new code. This allows for gradual rollouts, canary releases, and quick rollbacks.

    • A/B Testing & Experimentation:
      Run experiments by exposing different user segments to different features or variations, and measure the impact on user behavior.

    • Targeted Releases:
      Release features to specific users, groups, or regions for testing or phased rollouts.

    • Kill Switch:
      Instantly turn off problematic features in production with a single click.

    • Analytics & Monitoring:
      Track feature usage and experiment results to make data-driven decisions.

  • Alternatives: LaunchDarkly, Unleash, Optimizely

  • Example (JavaScript):

if (splitClient.getTreatment('new_feature') === 'on') {
  // Show new feature
} else {
  // Show old behavior
}
  • Summary:
    Split.io helps teams safely release, control, and measure features in production, reducing risk and enabling continuous delivery.

Eureka​

  • What: Service discovery tool (by Netflix).
  • Usage: Service discovery for microservices.
  • Key Features:
    • Registers and discovers services
    • Dynamic scaling and load balancing
    • REST API for registration and queries
  • Alternatives: Consul, Zookeeper, etcd

Akamai​

  • What: Leading cloud service provider for content delivery, security, and performance.

  • Key Features:

    • Global CDN (Content Delivery Network) for fast content delivery
    • Distributed Denial-of-Service (DDoS) protection and Web Application Firewall (WAF)
    • Edge computing and serverless functions
    • Real-time analytics and traffic management
    • Media delivery optimization (video streaming, downloads)
    • DNS and cloud security solutions
    • Akamai offers strong caching as part of its CDN services.
    • Cache Purge: Lets you quickly remove or update cached content worldwide.

  • Alternatives: Cloudflare, Fastly, Amazon CloudFront, StackPath

Cloudflare​

  • What: Cloud-based web performance and security platform.
  • Key Features:
    • Global CDN for faster content delivery
    • DDoS protection and WAF
    • DNS management with fast propagation
    • SSL/TLS encryption and automatic HTTPS
    • Bot management and rate limiting
    • Caching, image optimization, and edge computing (Cloudflare Workers)
    • Analytics and real-time traffic insights
  • Alternatives: Akamai, Fastly, Amazon CloudFront, StackPath

Apache HTTP Server​

  • What: Open-source web server software.
  • Key Features:
    • Serves static and dynamic web content
    • Highly configurable with modules (mod_rewrite, mod_ssl, etc.)
    • Virtual hosting (multiple sites on one server)
    • SSL/TLS support for secure connections
    • URL rewriting and access control
    • Logging, authentication, and proxy capabilities
    • Large community and extensive documentation
  • Alternatives: NGINX, Caddy, LiteSpeed, Microsoft IIS

Kong​

  • What: Open-source API gateway. An API gateway is a server that acts as a single entry point for client requests to multiple backend services or microservices. It handles routing, authentication, rate limiting, logging, and other common API tasks.
  • Usage: API gateway for managing, securing, and monitoring APIs.
  • Key Features:
    • Plugin-based extensibility
    • Authentication, rate limiting, logging
    • Supports REST, gRPC, WebSockets
  • Alternatives: Tyk, Apigee, Ambassador

Zuul​

  • What: Open-source edge service and API gateway (by Netflix).
  • Usage: Edge service and API gateway for microservices.
  • Key Features:
    • Dynamic routing and filtering
    • Load balancing and resiliency
    • Request/response transformation
  • Alternatives: Kong, NGINX, Traefik

Reverse Proxy​

  • What: Server that forwards client requests to backend servers.
  • Usage: Forwards client requests to backend servers, load balancing, SSL termination.
  • Key Features:
    • Load balancing
    • Caching and compression
    • Security (hides backend details)
  • Alternatives: NGINX, HAProxy, Apache HTTP Server

Explanation:

  • Requests to /dashboard are routed by the reverse proxy to localhost:3000.
  • Requests to /user-profile are routed to localhost:3001.

NGINX​

  • What: High-performance web server and reverse proxy. Pronounced "engine ex" (NGINX).
  • Usage: Web server, reverse proxy, and load balancer.
  • Key Features:
    • High performance and scalability
    • SSL/TLS termination
    • HTTP/2, WebSocket support
  • Alternatives: Apache HTTP Server, Caddy, HAProxy

JFrog Artifactory​

  • What: Universal artifact repository manager.
  • Usage: Universal artifact repository manager.
  • Key Features:
    • Stores and manages build artifacts (Docker, Maven, npm, etc.)
    • CI/CD integration
    • Access control and replication
  • Alternatives: Nexus Repository, GitHub Packages, AWS CodeArtifact

Pendo​

  • What: Product analytics and user feedback platform.
  • Usage: Product analytics and user feedback.
  • Key Features:
    • User behavior analytics and product usage insights
    • In-app guides, walkthroughs, and messaging
    • Feedback collection and surveys
    • Segmentation and targeting
    • Retention and adoption tracking
  • Alternatives: Mixpanel, Amplitude, WalkMe

Snowplow​

  • What: Open-source event analytics platform.
  • Usage: Collects, processes, and analyzes event-level data from websites, apps, and servers for advanced analytics.
  • Key Features:
    • Customizable data pipelines
    • Real-time and batch event tracking
    • Multi-source and multi-channel data collection
    • Integrates with data warehouses and BI tools
    • Full data ownership and flexibility
  • Alternatives: Segment, Google Analytics, RudderStack

Snowplow with Pendo for Analytics and In-App Guidance​

Snowplow and Pendo can be used together to provide a comprehensive analytics and user engagement solution:

  • Snowplow is ideal for collecting, processing, and analyzing detailed event-level data from your applications and websites. It gives you full control and ownership of your analytics data, enabling advanced reporting and integration with BI tools or data warehouses.

  • Pendo specializes in product analytics, in-app messaging, user feedback, and interactive guides or walkthroughs. It helps you understand user behavior, onboard users, and drive feature adoption directly within your app.

Combined Usage Example:​

  • Analytics:
    Use Snowplow to track every user interaction and funnel this data into your analytics stack for deep insights, segmentation, and custom reporting.
  • In-App Messaging & Guides:
    Use Pendo to create in-app messages, surveys, and step-by-step walkthroughs based on user behavior or segments identified from analytics (including data from Snowplow).
  • Personalization:
    Trigger Pendo guides or messages for specific user segments or behaviors detected by Snowplow event data.
  • Feedback Loop:
    Collect user feedback with Pendo and analyze the impact of product changes using Snowplow’s event data.

Summary:

  • Snowplow provides robust, customizable analytics and event tracking.
  • Pendo delivers targeted in-app messaging, onboarding, and user guidance.
  • Together, they enable data-driven product decisions and personalized user experiences.

Maven​

  • What: Build automation and dependency management tool for Java.
  • Usage: Build automation and dependency management for Java.
  • Key Features:
    • Declarative builds via pom.xml
    • Centralized dependency management
    • Convention over configuration (standard directory structure)
    • Large plugin ecosystem
    • Extensive documentation and community support
  • Best For: Projects needing stability, convention, and easy onboarding.
  • Alternatives: Gradle, Ant, SBT

Declarative means describing what you want to achieve, not how to do it. In programming and configuration, a declarative approach specifies the desired end state, and the system figures out the steps to reach that state.

Examples:

  • In a pom.xml (Project Object Model XML in Maven), you declare dependencies and plugins, not the exact commands to run.
  • In Terraform, you declare the infrastructure resources you want, not the step-by-step provisioning process.

Summary:
Declarative = describe the goal or outcome, not the procedure.

Gradle​

  • What: Modern build automation tool for Java, Kotlin, Groovy, and more.
  • Key Features:
    • Declarative and imperative build scripts (Groovy or Kotlin DSL)
    • Faster builds with incremental and parallel execution
    • Flexible and highly customizable
    • Supports multi-project builds
    • Rich plugin ecosystem
  • Best For: Projects needing performance, flexibility, and advanced build customization.
  • Alternatives: Maven, Ant, SBT

WildFly​

  • What: Open-source Java application server (formerly JBoss AS).
  • Usage: Java application server for deploying Java EE (Jakarta EE) applications.
  • Key Features:
    • Fast, modular, lightweight
    • Clustering and high availability
    • Web-based admin console
  • Alternatives: Apache Tomcat, GlassFish, Payara

Puma & Unicorn​

  • What: Concurrent web server for Ruby/Rails.
  • Usage: Web servers for Ruby on Rails applications.
  • Puma Key Features:
    • Multi-threaded, efficient, default for Rails, HTTP/2 support
  • What: Unicorn: Forking web server for Ruby/Rails.
  • Unicorn Key Features:
    • Multi-process, good for CPU-bound apps, simple and reliable
  • Alternatives: Passenger, Thin