Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Modern Software Engineering Practices

Table of Contents


Platform Engineering & Internal Developer Platforms

Platform engineering builds a self-service Internal Developer Platform (IDP) that abstracts infrastructure complexity so application teams can ship software autonomously. Think of it as the paved road: golden paths that are secure, compliant, and observable by default.

Key components of an IDP:

LayerPurposeExample
Backstage (Portal)Service catalog, documentation hubSpotify Backstage
Self-Service ActionsProvision environments, create reposGolden Path templates
Infrastructure AbstractionHide Kubernetes/VPC complexityCrossplane, Terraform CRDs
Observability StackBuilt-in metrics, logging, tracingOpenTelemetry, Grafana
GuardrailsEnforce policies without blockingOPA Gatekeeper, Kyverno

The difference from DevOps: DevOps is a culture; platform engineering is an engineering discipline that productizes DevOps capabilities into reusable, versioned internal products.

Developer Experience (DX) Principles

DX measures how easily developers can achieve their goals. Core principles:

  1. Fast feedback loops — sub-second compile, < 10 min CI, instant logs.
  2. Cognitive load reduction — consistent tooling, fewer choices (“batteries included”).
  3. Self-service — developers should never wait on a ticket to provision a database.
  4. Documentation as code — API docs, runbooks, and architectural decision records (ADRs) live in the repo.
  5. Local development paritydocker compose or DevContainers should mirror production closely enough that “works on my machine” is a solved problem.

Metrics to track: Time to first hello-world, deployment frequency, mean recovery time, and developer NPS surveys.

AI-Assisted Development

AI tools are reshaping every stage of the development lifecycle:

  • Code completion — Copilot, Cursor, Codeium. Interview angle: understand that these are next-token predictors over a context window; they hallucinate APIs, so code review is non-negotiable.
  • Code review — Automated PR summaries, security vulnerability detection (e.g., GitHub Advanced Security AI). AI can flag potential issues but should augment, not replace, human reviewers.
  • Testing — AI-generated unit tests increase coverage quickly but may test implementation details rather than behavior. Prefer generating tests against public interfaces.
  • Documentation — Auto-generated docstrings, README drafts, and ADR summaries.

Key risk: intellectual property leakage. Code sent to external LLM APIs may be retained and used in training. Enterprise deployments require self-hosted models or data-loss prevention (DLP) policies.

Software Supply Chain Security

The software supply chain is the path from source code to running binary. Attacks like SolarWinds and Codecov demonstrated that compromising any link in the cchain compromises the final product.

SBOM (Software Bill of Materials)

A machine-readable inventory of all components in a build:

  • SPDX and CycloneDX are the two dominant formats.
  • Required by US Executive Order 14028 for all software sold to the federal government.
  • Generated by tools: syft, Trivy, cdxgen.

SLSA (Supply-chain Levels for Software Artifacts)

A framework for ensuring artifact integrity across four levels:

LevelRequirementThreat Mitigated
1Documented build processTampering after build
2Hosted build platformTampering during build
3Hardened build + provenanceBuild platform compromise
4Hermetic + reproducible buildsAll upstream compromise

Provenance

Provenance is metadata cryptographically linking an artifact to its source, builder, and build parameters. Generated via SLSA GitHub Generator or Sigstore cosign.

Reproducible Builds & Dependency Pinning

A build is reproducible if building from the same source produces a bit-for-bit identical binary. This is hard: timestamps, random seeds, file ordering, and compiler versions all introduce non-determinism.

  • Dependency pinning — Lock files (package-lock.json, poetry.lock, go.sum) ensure exact versions. Use Dependabot or Renovate for automated, PR-based updates.
  • Deterministic Docker builds — Multi-stage builds with pinned base image digests (e.g., FROM node:20.11.0@sha256:... rather than tags).
  • Toolchain pinningasdf, nix, or .tool-versions lock the compiler/interpreter version.

Artifact Signing with Sigstore

Sigstore provides a free, open-source code-signing service designed for the modern supply chain:

  • cosign — Signs and verifies OCI container images using keyless signing (OIDC-based).
  • fulcio — Acts as a Certificate Authority, issuing short-lived certificates tied to your GitHub/GitLab identity.
  • rekor — A transparent, immutable log of all signatures (tamper-evident).

Workflow: After building an image, cosign sign attaches a signature. Admission controllers like Kyverno or OPA Gatekeeper verify signatures before allowing deployment to a Kubernetes cluster.

  • GitOps — Single source of truth in Git; declarative desired state reconciled by controllers.
  • Crossplane — Kubernetes-native infrastructure provisioning using CRDs (e.g., create an RDS instance via a PostgreSQLInstance custom resource).
  • CDK (Cloud Development Kit) — Define infrastructure in general-purpose languages (TypeScript, Python) rather than HCL/DSL.
  • Platform Orchestrators — Humanitec, Port, or Mia-Platform compose multiple IaC modules into golden path templates.
  • AIOps — ML-driven anomaly detection for capacity planning and incident response (Datadog, Dynatrace).

GitOps Evolution

GitOps has evolved from a deployment pattern to a full operating model:

GenerationApproachToolTrade-off
v1Manifest syncingArgoCD, FluxSimple but YAML-heavy
v2Kustomize/Helm overlaysHelm + FluxBetter composition, still imperative patches
v3Platform APIs + GitBackstage + CrossplaneApp-centric, infrastructure as Kubernetes resources

Core reconciliation loop: Git state (desired) → diff → Kubernetes API (actual) → converge.

Key principle: Pull-based deployments. Clusters pull their configuration from Git rather than receiving push-based deployments from CI, improving security posture.

Feature Flags & Progressive Delivery

Feature Flags

Feature flags decouple deployment from release:

  • Boolean flags — On/off toggle for a feature.
  • Percentage rollouts — Expose to 1%, 5%, 50% of users.
  • Targeting rules — By user segment, geography, or device.
  • Multivariate flags — A/B test multiple implementations.

Implementation: Use a dedicated service (LaunchDarkly, Unleash, Flipt) rather than homegrown solutions to avoid adding latency and complexity to your application.

Progressive Delivery

Progressive delivery extends feature flags with automated canary analysis:

  1. Deploy new version alongside stable (canary).
  2. Route a small percentage of traffic to canary.
  3. Observe SLOs (error rate, p99 latency) automatically.
  4. If metrics are healthy, increase traffic; otherwise, auto-rollback.

Tools: Argo Rollouts, Flagger, Istio (traffic splitting).

This is the operational backbone of trunk-based development — merge small, ship continuously, and let automated analysis decide when a feature is safe for 100%.


Interview Questions

  1. What is the difference between DevOps and platform engineering? DevOps is a cultural movement emphasizing collaboration between dev and ops. Platform engineering is an engineering discipline that productizes DevOps capabilities into self-service internal tools with paved roads, guardrails, and golden paths.

  2. How would you implement a feature flag system from scratch? Store flag configurations in a database. Serve via a low-latency API with caching (Redis or in-process cache with polling). Evaluate rules on the client or server side. Ensure flags add minimal latency (< 5ms) and have a fallback if the flag service is unavailable.

  3. What is an SBOM and why does it matter? A Software Bill of Materials is a machine-readable list of all components (libraries, frameworks, transitive dependencies) in a software artifact. It enables rapid vulnerability assessment when a new CVE is published (e.g., Log4Shell).

  4. Explain SLSA Level 3 requirements. Level 3 requires a hardened, non-falsifiable build process with generated provenance. The provenance cryptographically links the output artifact to the source repo, builder identity, and build parameters. This mitigates build platform compromise.

  5. How does Sigstore keyless signing work? Sigstore uses OIDC tokens (e.g., from GitHub Actions) to authenticate the signer to Fulcio, which issues a short-lived certificate. The artifact is signed with cosign, and the signature is recorded in Rekor’s transparency log. Verification checks the OIDC identity against a policy.

  6. What is the difference between canary deployment and blue-green deployment? Blue-green deploys a full parallel environment and switches traffic atomically. Canary deploys the new version alongside the old with gradual traffic shifting. Canary provides more granular feedback and lower blast radius but is more complex to implement.

  7. How do you measure developer experience? Quantitative: deployment frequency, lead time for changes, MTTR, change failure rate (DORA metrics). Qualitative: developer NPS surveys, onboarding time-to-first-deploy, friction audits (counting steps to provision an environment).

  8. What risks does AI-assisted development introduce? Code hallucination (invented APIs), over-reliance reducing deep understanding, IP leakage to external APIs, and biased or insecure generated code. Mitigation: mandatory code review, self-hosted models for sensitive codebases, and treating AI output as a draft, not final.

References