Question 1

Your team wants to spin up on-demand preview environments in Kubernetes for every open pull request. How do you design the CI/CD pipeline to handle namespace creation, DNS record registration, dynamic ingress routing, and automated teardown upon PR closure to avoid orphaned infrastructure?

Accepted Answer

Design: a PR-opened webhook triggers a CI job that creates a namespace named `pr-<number>`, applies a Helm chart or Kustomize overlay with a unique subdomain, lets cert-manager issue a certificate, has external-dns create a DNS record pointing at the ingress load balancer, and the ingress controller adds a host-based routing rule. On every PR sync, redeploy into the same namespace. On PR close/merge, a teardown job runs `helm uninstall` and `kubectl delete namespace`, which cascades and deletes everything inside it; external-dns automatically removes the DNS record once it sees the ingress object disappear.

The part people forget is the safety net: webhooks get missed (a CI runner crash mid-teardown, a dropped delivery), so you also run a nightly CronJob that lists namespaces labeled `ephemeral=true` and deletes any whose creation timestamp is older than 48 hours, catching orphans the event-driven path missed.

```yaml
# GitHub Actions trigger
on:
  pull_request:
    types: [opened, synchronize, closed]
jobs:
  deploy:
    if: github.event.action != 'closed'
    steps:
      - run: kubectl create namespace pr-${{ github.event.number }} --dry-run=client -o yaml | kubectl apply -f -
      - run: helm upgrade --install pr-${{ github.event.number }} ./chart -n pr-${{ github.event.number }}
  teardown:
    if: github.event.action == 'closed'
    steps:
      - run: helm uninstall pr-${{ github.event.number }} -n pr-${{ github.event.number }}
      - run: kubectl delete namespace pr-${{ github.event.number }}
```

Question 2

Integrating SAST and container vulnerability scanners into the primary CI pipeline causes 40% of PR builds to fail due to low-severity findings and false positives, frustrating developers. How do you design an automated security gating and triage policy that blocks critical CVEs without slowing down development velocity?

Accepted Answer

Use a tiered severity gate instead of an all-or-nothing scan: block the PR merge only on Critical/High CVEs that also have a known fix available and ideally exploit signal (an EPSS score or presence on CISA's Known Exploited Vulnerabilities list) — report Medium/Low findings as a non-blocking PR comment/annotation, tracked in a backlog with an SLA instead of failing the build. Never fail on a vulnerability with no available patch, since blocking developers for something they can't currently fix just trains them to ignore the gate.

Maintain a suppression file (`.trivyignore` or equivalent) for accepted false positives, each entry with an owner and an expiry date so it gets re-reviewed instead of silently living forever. Run a fast SAST tool (like Semgrep) inline on just the diff's changed files for quick PR feedback, and push the full deep scan to an async nightly job that doesn't block anyone's merge.

```bash
trivy image --severity CRITICAL,HIGH --ignore-unfixed --exit-code 1 myapp:sha
```
`--ignore-unfixed` and the severity filter mean the build only fails for high-severity issues that actually have a patch available; everything else is printed for visibility but doesn't block the merge.

Question 3

An integration test suite takes 45 minutes to execute in CI. How do you implement intelligent test sharding across dynamic runners, distributed test caching, and an automated flaky-test quarantine mechanism to bring pipeline execution time under 8 minutes?

Accepted Answer

Split tests across parallel runners by historical execution-time balance rather than alphabetically or by file count — a test-splitting tool (CircleCI's `--split-by=timings`, or a JSON file of last-run durations) assigns each shard roughly equal wall-clock work instead of one shard getting all the slow tests. Pair that with a distributed build/dependency cache keyed on the lockfile hash so unrelated shards don't each redundantly reinstall dependencies or rebuild unchanged packages.

For flakiness, track each test's pass/fail history in a datastore; a failing test gets one automatic retry in isolation, and if it flip-flops across recent runs it gets tagged "quarantined" — pulled out of the main gating suite into a separate non-blocking job that still runs and alerts the owner, but never blocks merges on its own.

```yaml
# 8-way sharded matrix
strategy:
  matrix:
    shard: [1,2,3,4,5,6,7,8]
steps:
  - run: test-splitter run --total-shards=8 --shard-index=${{ matrix.shard }}
```
Together this turns a 45-minute serial run into roughly 45/8 ≈ 6 minutes per shard, plus a separate always-green-required quarantine job running the historically flaky subset out of band.

Question 4

To eliminate static cloud access keys stored in CI/CD secrets managers, design a secretless authentication architecture using OpenID Connect (OIDC) and Workload Identity Federation that grants ephemeral, scoped IAM permissions during pipeline execution.

Accepted Answer

Configure the cloud provider (AWS IAM's OIDC identity provider, or GCP Workload Identity Federation) to trust the CI platform's own OIDC token issuer (for GitHub Actions, `token.actions.githubusercontent.com`). Create an IAM role whose trust policy is conditioned on claims inside that token — typically the repo and branch — so only builds matching that exact condition can assume it. At pipeline runtime, the CI job requests a short-lived signed JWT from its own platform, hands it to the cloud provider's STS endpoint, and exchanges it for temporary credentials (commonly 15–60 minutes), with no static key ever stored anywhere.

The security win is that even if the CI logs or a compromised job leak the credentials, they expire almost immediately and were scoped to one specific repo/branch/role in the first place — nothing like a long-lived AWS access key sitting in a secrets manager for months.

```yaml
permissions:
  id-token: write
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/ci-deploy-role
      aws-region: ap-south-1
```
The role's trust policy has a condition like `StringEquals: {"token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"}`, so only main-branch builds of that specific repository can ever assume it.

Question 5

Design an automated release pipeline that achieves SLSA Level 3 compliance by generating signed cryptographic provenance attestations and Software Bills of Materials (SBOMs) using Sigstore and Cosign during container compilation.

Accepted Answer

Use Cosign's keyless signing mode: Sigstore's Fulcio CA issues a short-lived certificate bound to the CI job's own OIDC identity (no long-term signing key to manage or leak), and the signature is recorded in Rekor's public transparency log so it's independently auditable later. At build time, generate an SBOM (via `syft` or `docker buildx sbom`) listing every dependency in the image, and generate an in-toto provenance attestation describing the exact build parameters, source commit, and builder identity — then attach both to the image manifest as signed attestations with `cosign attest`.

At deploy time, an admission controller (Kyverno, or Sigstore's own policy-controller) verifies the signature and provenance before allowing the image to run at all, rejecting anything unsigned or with a broken chain of custody — this is the enforcement step that actually makes the provenance meaningful rather than just decorative.

```bash
cosign sign --keyless myregistry/app@sha256:abcd...
cosign attest --predicate sbom.json --type spdx --key cosign.key myregistry/app@sha256:abcd...
```
A Kubernetes admission policy then blocks any image lacking a valid Rekor-logged signature, satisfying SLSA L3's requirement that provenance be non-forgeable and verified prior to deployment.

Question 6

Your build farm needs to compile multi-architecture container images for both AMD64 and ARM64. QEMU emulation in CI takes 4x longer than native compilation. How do you architect native cross-compilation worker nodes using Docker Buildx and distributed remote builders to accelerate build times?

Accepted Answer

QEMU (via binfmt_misc) is slow specifically because it translates every ARM64 instruction one at a time on an x86 host — instead, provision real native ARM64 machines (AWS Graviton instances, or ARM-hosted CI runners) alongside your existing AMD64 runners, and use Docker Buildx's multi-node builder to fan a single build command across both, each compiling natively for its own architecture in parallel.

Buildx then assembles both digests into one multi-arch manifest list, so consumers pulling `myapp:latest` automatically get the image matching their own platform without knowing multiple architectures even exist.

```bash
docker buildx create --name multiarch --append --node amd64-node ssh://ci-amd64
docker buildx create --name multiarch --append --node arm64-node ssh://ci-arm64
docker buildx build --builder multiarch --platform linux/amd64,linux/arm64 -t myapp:latest --push .
```
Because each architecture builds on hardware that actually matches it, total wall-clock time drops to roughly `max(amd64_time, arm64_time)` running in parallel, instead of `amd64_time + 4×arm64_time` when ARM64 was emulated serially on an x86 box.

Question 7

A security audit mandates that all production containers must run in rootless mode with user namespaces enabled. How do you refactor base images, handle volume permission re-mapping, and configure containers requiring privileged port bindings below 1024?

Accepted Answer

Rebuild base images to run as a non-root UID (`USER 1000` in the Dockerfile) and enable user-namespace remapping in the runtime (Docker's `userns-remap`, or rootless Podman/containerd) so container UID 0 maps to an unprivileged host UID — even a full container-root compromise can't touch real host root. For bind-mounted volumes, which keep their original host ownership, add an init step (an init container in Kubernetes, or a `chown` in the entrypoint) that fixes ownership to match the remapped UID range before the app starts.

For ports below 1024, which the kernel normally reserves for root, don't run as root at all — grant just the one capability the binary actually needs.

```bash
setcap 'cap_net_bind_service=+ep' /usr/bin/myapp
```
```yaml
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    add: ["NET_BIND_SERVICE"]
```
The process binds port 80 while running fully as an unprivileged user, satisfying the audit without needing root anywhere in the container.

Question 8

A microservice Dockerfile rebuilds its entire Node/Python dependency layer on every single commit because source files are copied prior to package manifest installation. How do you restructure multi-stage layers and utilize BuildKit remote cache registries to maximize cache hits across CI runs?

Accepted Answer

Docker caches each layer by the hash of its inputs, so if you `COPY . .` before installing dependencies, any source change invalidates the dependency-install layer too, forcing a full reinstall every commit. Reorder so the manifest (package.json/requirements.txt) is copied and installed first, and the rest of the source is copied afterward — a source-only commit then hits the cache for the dependency layer instead of rebuilding it.

On CI runners without persistent local disk, push the BuildKit cache to a registry so cache hits survive across ephemeral machines, not just on one long-lived box.

```dockerfile
FROM node:20
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
```
```bash
docker buildx build \
  --cache-to=type=registry,ref=myrepo/cache \
  --cache-from=type=registry,ref=myrepo/cache \
  -t myrepo/app:latest --push .
```
Most commits now only touch the `COPY . .` layer onward, only rebuilding the `npm ci` layer when `package.json` itself actually changes.

Question 9

Containers running in Kubernetes fail to terminate cleanly within the termination grace period and accumulate zombie processes in production. Explain how PID 1 process responsibilities and signal forwarding work inside Linux containers, and how wrapping entrypoints with lightweight init systems resolves process reaping.

Accepted Answer

PID 1 in Linux has special kernel semantics: it doesn't inherit the normal default signal dispositions, so if your app process is PID 1 and doesn't explicitly register a SIGTERM handler, the signal is simply ignored. Kubernetes sends SIGTERM on shutdown and waits out the grace period, then falls back to SIGKILL — so an app that never registered a handler always gets abruptly force-killed instead of shutting down gracefully. PID 1 is also responsible for reaping zombie child processes (calling `wait()` on them); if your app spawns subprocesses without being written as a proper init, those zombies pile up and consume process table entries.

The fix is to not let the application be PID 1 at all — wrap it with a minimal init process like `tini`, which becomes PID 1 itself, correctly forwards signals to the real app, and reaps orphaned children on its behalf.

```dockerfile
ENTRYPOINT ["tini", "--", "node", "server.js"]
```
Now SIGTERM from kubelet reaches `node.js` properly through tini's forwarding, and any zombie children the app spawns get reaped automatically, resolving both the abrupt-kill and zombie-accumulation problems at once. (`docker run --init` and Kubernetes' `shareProcessNamespace` sidecar pattern solve the same problem in slightly different ways.)

Question 10

A containerized data processing job crashes intermittently with a bus error due to the default 64MB Linux shared-memory limit. How do you safely expand shared memory in container runtime configurations and Kubernetes volume mounts without exposing host memory?

Accepted Answer

Containers default to a 64MB `/dev/shm` tmpfs, and libraries that use shared memory heavily (NumPy multiprocessing, PyTorch DataLoader workers, some databases) can exceed that and trigger a SIGBUS bus error the moment they write past it. In plain Docker this is one flag: `docker run --shm-size=2g myimage`.

In Kubernetes there's no equivalent single flag — `/dev/shm` has to be explicitly mounted as a memory-backed `emptyDir` volume with an explicit size cap, so a runaway process can't silently consume unbounded host RAM.

```yaml
volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 2Gi
containers:
  - name: worker
    volumeMounts:
      - mountPath: /dev/shm
        name: dshm
```
This tmpfs is backed by RAM but bounded by `sizeLimit`, and that usage counts against the Pod's own memory request/limit — so it's visible to the scheduler and can't be used to quietly overrun the node's memory outside Kubernetes' accounting.

Question 11

Multiple automated CI pipelines executing concurrent Terraform plans encounter persistent state lock acquisition errors on a DynamoDB lock table. How do you design backend state partitioning, lock timeout retries, and pipeline orchestration to prevent pipeline concurrency deadlocks?

Accepted Answer

The first fix is architectural, not a retry setting: partition state by blast radius — one state file per environment/service/region instead of one giant shared state — so unrelated pipelines never even contend for the same lock. For state that genuinely can be touched concurrently (two engineers applying the same module), rely on the backend's native locking (S3 backend + DynamoDB lock table, or Terraform Cloud's built-in locking) with a bounded wait rather than an instant failure.

```bash
terraform apply -lock-timeout=5m
```
Beyond the lock itself, serialize applies against a given state at the orchestration layer so contention rarely happens in the first place — a CI concurrency group ensures only one apply against a given state runs at a time.

```yaml
concurrency:
  group: tf-prod-network
  cancel-in-progress: false
```
With smaller, more numerous state files and a queue that prevents overlapping applies, the DynamoDB lock table becomes a safety net for the rare edge case rather than something pipelines are routinely fighting over.

Question 12

You need to refactor a monolithic Terraform configuration into separate nested reusable modules managing hundreds of active production resources. How do you utilize moved blocks and state manipulation commands to reorganize the state tree without triggering resource destructions or recreation?

Accepted Answer

`moved` blocks tell Terraform "this resource used to live at address X, it now lives at address Y," so it updates the state in place instead of planning a destroy-then-create when the resource's address changes due to a refactor.

```hcl
moved {
  from = aws_instance.web
  to   = module.web_servers.aws_instance.web
}
```
After adding the moved blocks for every relocated resource, run `terraform plan` — it should show zero unintended changes, confirming Terraform mapped the old state to the new address rather than proposing to tear anything down. For very large one-time refactors, `terraform state mv` does the same thing imperatively from the CLI before the code even has moved blocks, useful as a quick manual fix, but moved blocks are preferred for anything checked into version control since they're self-documenting and anyone re-running the same plan later gets the identical safe mapping instead of having to remember to run a state-mv command by hand.

Question 13

Design an automated GitOps-driven IaC pipeline that continuously detects configuration drift against live cloud infrastructure, alerts engineers on unmanaged manual changes, and triggers automated safe reconciliation workflows.

Accepted Answer

Run a scheduled job (nightly, or on a CI cron trigger) that executes `terraform plan -detailed-exitcode` against every managed state — exit code 0 means no drift, 2 means drift was detected, 1 means an error occurred. On exit code 2, parse the plan's JSON output and post a summary naming exactly which resources and attributes drifted to Slack/PagerDuty, plus open a tracking ticket automatically.

For "safe automated reconciliation," only auto-apply changes on an allowlist of clearly safe, additive, non-destructive attribute types (like a manually-added tag) — anything touching deletion or a security-relevant field (IAM policy, security group rule, encryption setting) routes to a human for approval instead, since drift is sometimes a deliberate manual emergency fix that shouldn't get silently reverted by automation.

```bash
terraform plan -detailed-exitcode -out=plan.tfplan
if [ $? -eq 2 ]; then
  terraform show -json plan.tfplan | ./notify-drift.sh
fi
```
Dedicated drift tools like `driftctl`, or Terraform Cloud's built-in drift detection, already package this scan-and-diff loop rather than needing to hand-build the scheduling and parsing yourself.

Question 14

How do you integrate Open Policy Agent (OPA) and Conftest into a pre-merge CI validation pipeline to parse Terraform plan JSON files, enforce strict mandatory cost-allocation tagging, and reject non-compliant storage encryption settings?

Accepted Answer

Generate the plan as JSON in CI (`terraform show -json tfplan > plan.json`), then run Conftest against it using Rego policies that inspect the planned resource changes before anything is actually applied.

```rego
package main

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  not resource.change.after.tags["cost-center"]
  msg := sprintf("S3 bucket %s missing cost-center tag", [resource.address])
}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  resource.change.after.server_side_encryption_configuration == null
  msg := sprintf("S3 bucket %s has no encryption configured", [resource.address])
}
```
```bash
conftest test plan.json -p policy/
```
Any firing `deny` rule fails the CI job and blocks the PR from merging until the tag or encryption setting is fixed — pushing org-wide compliance enforcement into code review, before deployment, instead of catching it afterward via a cloud security scanner.

Question 15

You are provisioning a cloud architecture where an IAM role requires an S3 bucket ARN, but the S3 bucket policy requires the IAM role ARN, creating a circular dependency in Terraform. How do you resolve this dependency cycle using standalone attachment resources and multi-phase module bootstrapping?

Accepted Answer

Break the cycle by not referencing either resource from inside the other at all — create the IAM role and the S3 bucket independently, with no cross-reference, then wire the coupling through separate standalone "attachment" resources that each reference both ARNs and naturally get created last, once both primaries already exist.

```hcl
resource "aws_iam_role" "app" { ... }
resource "aws_s3_bucket" "data" { ... }

resource "aws_iam_role_policy" "app_s3_access" {
  role   = aws_iam_role.app.id
  policy = jsonencode({
    Statement = [{ Effect = "Allow", Action = "s3:*", Resource = aws_s3_bucket.data.arn }]
  })
}

resource "aws_s3_bucket_policy" "allow_app_role" {
  bucket = aws_s3_bucket.data.id
  policy = jsonencode({
    Statement = [{ Effect = "Allow", Principal = { AWS = aws_iam_role.app.arn }, Action = "s3:GetObject" }]
  })
}
```
Since neither `aws_iam_role.app` nor `aws_s3_bucket.data` references the other, Terraform can create both in either order, and the two attachment resources only run once both ARNs are already resolved. For genuinely separate bootstrap phases (e.g., cross-account trust that must pre-exist before the second account can even be planned), split into two sequential applies/workspaces, passing the first apply's outputs as the second's input variables.

Question 16

During a blue-green release of a real-time messaging application, switching router traffic to the green deployment abruptly terminates 500,000 active WebSocket connections. How do you design connection draining, graceful client reconnection backoff, and dual-routing mechanisms during blue-green switchovers?

Accepted Answer

Don't cut traffic over instantly — put both blue and green behind a router capable of gradual traffic shifting (an L7 load balancer or service mesh, not a DNS swap) and route only *new* connections to green, leaving blue's already-open WebSocket connections alone. Blue then enters a draining state: it stops accepting new connections but keeps serving existing ones for a bounded window (say 10–15 minutes), while proactively telling each connected client over its existing socket to reconnect soon.

Crucially, that reconnect signal carries jittered backoff so all 500,000 clients don't reconnect in the same instant and stampede green — each client picks its own random delay within a window.

```json
{"type": "reconnect", "delaySec": 137}
```

Only after the drain window elapses (or blue's open-connection count hits zero) does blue actually get terminated. In infrastructure terms this looks like an Envoy/service-mesh listener drain, which stops routing new connections to blue's listener while letting open ones finish naturally, combined with the app layer pushing that jittered reconnect message — spreading reconnects over minutes instead of the instant spike a hard cutover would cause.

Question 17

Architect an automated progressive canary deployment pipeline using a service mesh canary controller and Prometheus where traffic is shifted in 10% increments every 5 minutes, automatically executing an immediate rollback if the p99 latency SLO breaches 200ms or 5xx error rate exceeds 0.5%.

Accepted Answer

Use a canary controller like Flagger or Argo Rollouts wired to your service mesh (Istio/Linkerd) for traffic splitting and to Prometheus for the actual health signal. Define the rollout as a sequence of steps: shift 10% more traffic to the canary, wait 5 minutes, query Prometheus over that window for p99 latency and 5xx rate, and if either breaches its threshold, automatically revert all traffic to the stable version and abort the rollout — no human has to notice and react fast enough.

```yaml
analysis:
  interval: 5m
  stepWeight: 10
  maxWeight: 100
  metrics:
    - name: latency-p99
      thresholdRange: { max: 200 }
      interval: 5m
    - name: error-rate
      thresholdRange: { max: 0.5 }
      interval: 5m
```
Flagger runs these PromQL-backed checks at every step and adjusts the mesh's traffic-split resource (an Istio `VirtualService` weight) directly, snapping the canary weight back to 0% the instant a threshold trips — the whole progressive-rollout-plus-guardrail loop runs unattended, with the rollback happening in the same automated cycle that detected the problem.

Question 18

You are deploying breaking database migrations across a phased multi-ring deployment (Internal Canary -> 10% Ring -> 100% Ring). How do you structure the expand-and-contract migration pattern in your deployment automation so that legacy and updated application versions run concurrently across rings without data loss?

Accepted Answer

Expand-and-contract splits a breaking schema change into steps that are each backward-compatible on their own. Expand: add the new column/table alongside the old one, and deploy app code that writes to *both* old and new fields but still reads from the old one — this version is safe running next to the pre-migration version. Migrate: backfill historical rows into the new schema. Contract phase 1: deploy app code that reads from the new field (falling back to the old one if it's missing) while still writing both. Only once every ring is fully on a version that no longer needs the old field do you run Contract phase 2 and actually drop it.

Because each step is compatible with the step immediately before and after it, the Internal Canary, the 10% Ring, and the 100% Ring can each sit on a *different* app version at the same moment without either version breaking against the shared database — nothing requires all rings to migrate in lockstep.

Example: adding a `full_name` column while keeping `first_name`/`last_name` — expand adds the column and dual-writes both representations, a backfill job populates `full_name` for existing rows, and only once 100% of rings read `full_name` does a later, separate migration finally drop the two old columns.

Question 19

Over two years, an application has accumulated 200+ stale feature flags, increasing codebase technical debt and test permutation complexity. How do you establish an automated lifecycle management system that tracks flag staleness, warns flag owners, and opens automated cleanup pull requests?

Accepted Answer

Track metadata at flag creation time: owner, creation date, and an intended lifetime — "temporary" flags default to something like a 90-day TTL, while genuinely permanent ops flags (kill switches, region toggles) are explicitly exempted. A scheduled job then checks each flag's real evaluation data from the flag platform (LaunchDarkly, Unleash, etc.) — if a flag has sat at 100% (or 0%) for N consecutive days with zero variation, that's a strong signal it's no longer deciding anything and is just dead conditional code wrapped around a fixed outcome.

That job posts a reminder to the owner in Slack; if it goes unaddressed through a second reminder, it auto-opens a PR that removes the flag's conditional (keeping only the winning branch) and deletes the flag from the platform, for the owner to simply review and merge.

This converts flag cleanup from "someone eventually remembers to do a sweep" into a routine, low-effort backlog item that surfaces on its own — instead of relying on a person periodically auditing a spreadsheet of 200+ flags, which is exactly how the backlog got to 200 in the first place.

Question 20

What is DevOps, and how is it different from traditional siloed IT operations?

Accepted Answer

DevOps is a set of practices and a culture that merges software development and IT operations so the same team (or tightly collaborating teams) builds, ships, and runs software, instead of a dev team throwing a finished build "over the wall" to a separate ops team that never wrote it. Traditional IT was siloed: developers wrote code and handed it off, operations deployed and kept it running, and the two groups had misaligned incentives (dev wants to ship features fast, ops wants stability) — that handoff created blame when things broke and operational knowledge that never fed back into how code was written.

DevOps closes that loop with shared ownership, automation (CI/CD, Infrastructure as Code), and continuous feedback where monitoring data directly informs what the development team fixes next.

Example: instead of a dev team filing a ticket for ops to manually provision a server once a quarter, a DevOps team owns a pipeline where a merged pull request automatically builds, tests, and deploys itself within minutes — and the engineers who wrote the code are also the ones who get paged if it breaks, which pushes them to write more operable, observable code from the start.

Question 21

What is CI/CD, and what's the difference between continuous delivery and continuous deployment?

Accepted Answer

Continuous Integration means developers merge code into a shared branch frequently — multiple times a day — and every merge automatically triggers a build and test run, catching integration problems within minutes instead of weeks after a large branch finally merges. Continuous Delivery builds on that: once CI passes, the pipeline automatically produces a release-ready artifact that could be deployed to production at any time, but a human still has to click the button. Continuous Deployment removes that button entirely — any change that clears every automated gate (tests, security scans, canary health checks) goes straight to production with no manual approval step at all.

Example: a team on continuous delivery has every merge to `main` automatically build, test, and sit ready behind a "Deploy to Prod" button in their pipeline dashboard; a team on continuous deployment has that exact same pipeline skip the button and push straight to production, relying entirely on its automated gates — and often automated canary rollback — to catch anything a human review would have caught.

Question 22

What is the difference between a Docker container and a virtual machine?

Accepted Answer

A virtual machine virtualizes hardware: a hypervisor runs several full guest operating systems, each with its own kernel, on top of one host — so every VM carries the overhead of an entire OS (gigabytes of disk, real memory reserved for its own kernel, boot times measured in tens of seconds to minutes). A container virtualizes the operating system instead: every container on a host shares that host's single kernel, and isolation comes from kernel features — namespaces for process/network/filesystem separation, cgroups for resource limits — rather than a separate kernel per workload. That makes containers far lighter: megabytes instead of gigabytes, startup in milliseconds to a couple seconds, and you can pack far more of them onto the same hardware.

Example: running 20 instances of a small web service as VMs means 20 separate kernels each reserving their own baseline memory just to exist; running the same 20 as Docker containers shares one kernel and only pays for each app process itself. The trade-off is isolation strength — a container escape is more dangerous than a VM escape since it's closer to the shared host kernel, which is why security-sensitive multi-tenant platforms often run containers *inside* lightweight VMs (Firecracker, gVisor) for defense in depth.

Question 23

What is Kubernetes, and what problem does it solve?

Accepted Answer

Kubernetes is a container orchestration platform: given a fleet of machines (nodes), it schedules containers onto them, keeps your declared number of replicas of each service running — restarting or rescheduling a container if it crashes or its node dies — handles service discovery and load balancing between services, and manages rolling out new versions or rolling back bad ones. You describe the desired state in a YAML manifest ("run 5 replicas of this image, expose it on this port"), and Kubernetes' control loop continuously reconciles the actual cluster toward that declared state.

The problem it solves: running containers on a single Docker host doesn't survive that host dying, doesn't scale automatically, and gives you no clean way to do a zero-downtime rolling update across many machines by hand.

Example: a Deployment declaring `replicas: 5` — if a node hosting 2 of those Pods crashes, Kubernetes' scheduler notices the pod count dropped below the declared 5 and reschedules 2 new Pods onto healthy nodes automatically, with no human intervention required to restore capacity.

Question 24

What is Infrastructure as Code, and why is it preferred over manual provisioning?

Accepted Answer

Infrastructure as Code means defining servers, networks, databases, and other cloud resources in version-controlled configuration files (Terraform, CloudFormation, Pulumi) instead of clicking through a cloud console or running one-off CLI commands by hand. That gives infrastructure the same benefits code review already gives application code: every change is a diff someone reviews before it happens, there's a git history of exactly what changed and why, and running `terraform plan` shows exactly what will change before it happens, instead of discovering the effect of a console click after the fact.

Manual provisioning ("click-ops") is undocumented and unreviewable — if the person who configured a setting leaves, or an environment needs recreating after a disaster, nobody can reliably reconstruct it.

Example: `resource "aws_instance" "web" { instance_type = "t3.medium", ami = var.ami_id }` checked into git means rebuilding an entire environment after a region-wide outage is a `terraform apply` against a new region, not days spent trying to remember every console setting configured eighteen months earlier.

Question 25

What's the difference between a blue-green deployment and a canary deployment?

Accepted Answer

Blue-green deployment runs two full, identical production environments — "blue" currently live, "green" the new version — and once green is fully deployed and verified, you switch all traffic to it at once, typically via a router or load-balancer swap, keeping blue on standby for an instant rollback. Canary deployment instead sends only a small slice of live traffic (say 5%) to the new version while most traffic still hits the old one, watches real production metrics on that slice, and only increases the percentage gradually if things look healthy — it never does a hard, all-at-once cutover.

The trade-off: blue-green gives a clean, instant rollback (flip the router back) but doesn't limit blast radius during the switch itself — if a bug only shows up under real production load, 100% of users hit it the moment you cut over. Canary limits blast radius to a small fraction of users if something's wrong, at the cost of needing more sophisticated traffic-splitting infrastructure and taking longer to reach full rollout.

Example: a real-time payments API would likely prefer canary specifically because a bad deploy hitting 100% of traffic instantly is unacceptable, even though it's slower to fully roll out than a blue-green swap.

Question 26

In Kubernetes, what's the difference between a Pod, a Deployment, and a Service?

Accepted Answer

A Pod is the smallest deployable unit in Kubernetes: one or more containers that always run together on the same node, sharing a network namespace and optionally storage volumes — you rarely create a bare Pod in production because it has no self-healing; if it dies, it's simply gone. A Deployment is a higher-level controller that manages a set of identical Pods: you declare "I want 3 replicas of this spec," and it creates, replaces, and reschedules Pods to keep that count satisfied, and also manages rolling updates that gradually swap old-version Pods for new ones. A Service is a stable network identity — a fixed virtual IP and DNS name — that load-balances traffic across whichever Pods currently match its label selector, since Pods themselves get new IPs every time they're recreated.

Example: a Deployment `frontend` manages 3 Pods labeled `app=frontend`; a Service `frontend-svc` with `selector: app=frontend` gives those Pods one stable DNS name that other services call. When the Deployment replaces a Pod during a rolling update, the Service automatically updates which Pod IPs it routes to — callers never notice.

Question 27

How does a load balancer decide which backend server to send a request to?

Accepted Answer

A load balancer sits in front of multiple backend instances and distributes requests among them by a chosen algorithm. Round-robin cycles through the backend list in order regardless of current load — simple, but can overload a server that's already slow or busy. Least-connections tracks each backend's active connection count and routes new requests to whichever has the fewest — better when request durations vary a lot. Weighted algorithms bias distribution toward specific backends (e.g., a bigger instance type gets 2x the traffic), and consistent hashing routes based on a hash of some request property like client IP or session ID, so the same client reliably lands on the same backend — useful for sticky sessions or cache locality.

Beyond the routing algorithm itself, most load balancers also run active health checks against each backend and immediately stop routing to any that fail — which is a core part of how zero-downtime deploys and automatic self-healing actually work: a crashed instance is silently pulled from rotation without any DNS change or human intervention.

Question 28

What does "immutable infrastructure" mean, and why do teams adopt it?

Accepted Answer

Immutable infrastructure means once a server, container, or VM is deployed, you never modify it in place — no SSH-ing in to patch a config or upgrade a package on a running instance. Instead, any change (new app version, OS patch, config update) gets baked into a brand-new image, and you replace the old instance with a fresh one built from that image, then destroy the old one. The alternative, mutable infrastructure, patches servers in place over months, accumulating "configuration drift" — ad-hoc manual fixes nobody fully tracked, until no one can reliably say what state a given server is actually in or reproduce it from scratch.

Teams adopt it because deployments become deterministic and rollback trivial — the old image is still sitting there, just redeploy it — and it eliminates an entire class of "works on this server but not that one" bugs, since every running instance at any moment came from the exact same image.

Example: instead of `ssh` and `apt upgrade` on a live web server, a new AMI is built with the patch baked in via Packer, new instances deploy from it behind the load balancer, and once health checks pass the old instances are terminated — with the old AMI still available for an instant rollback if the new one misbehaves.

Question 29

What's the difference between monitoring and observability?

Accepted Answer

Monitoring is watching a predefined set of metrics and dashboards for conditions you already anticipated — CPU above 80%, error rate above 1%, disk nearly full — and it answers "is something wrong?" only for the things you thought to measure ahead of time. Observability is a broader property: how well can you ask arbitrary *new* questions about a system's internal state from the outside, using its outputs (metrics, logs, and traces together), especially about failure modes nobody predicted in advance. A highly observable system lets you debug "why did this one specific request take 4 seconds at 3:14pm" even though nobody built a dashboard for that exact scenario.

Example: monitoring is the alert that fires saying "checkout error rate spiked to 12% at 3:14pm." Observability is what lets you then drill from that alert into a distributed trace for an affected request, see it hit a slow downstream fraud-check API call that started timing out, and correlate that with a log line showing the fraud API's TLS certificate had just expired — a root cause nobody wrote a dashboard for, discoverable only because traces, logs, and metrics were correlated by a shared request ID.

Question 30

What's the difference between Git trunk-based development and Gitflow?

Accepted Answer

Gitflow uses long-lived branches for different purposes — `develop` for ongoing integration, feature branches per feature, `release` branches to stabilize before shipping, and `main` receiving only finished releases — with merges happening at defined checkpoints. It suits teams doing scheduled, versioned releases, but tends to let branches drift far apart, producing large, risky merges and slow feedback on integration conflicts. Trunk-based development instead has everyone commit small, frequent changes directly to a single shared trunk, using short-lived feature branches (hours to a couple days, not weeks) merged behind feature flags if the feature isn't finished yet — integration problems surface within hours, not after weeks of divergence.

Trunk-based development is the standard fit for continuous delivery/deployment because it keeps `main` always close to what's about to ship; Gitflow's release-branch model is harder to reconcile with deploying many times a day, since it's built around batching changes into discrete numbered releases rather than a constant stream of small ones.

Question 31

What's the practical difference between Ansible, Chef, and Puppet?

Accepted Answer

All three are configuration management tools that declare a server's desired state (packages installed, files present, services running) in code instead of manually SSH-ing in and running commands — but they differ in architecture. Puppet and Chef are agent-based: a persistent agent runs on every managed node and periodically pulls its configuration from a central master, giving strong continuous drift correction (a manual change gets reverted on the next agent run) at the cost of maintaining that agent infrastructure. Puppet uses a declarative DSL; Chef uses Ruby-based "recipes," giving it more imperative flexibility at the cost of being harder to reason about declaratively.

Ansible is agentless: it connects over plain SSH and pushes changes on demand using YAML playbooks, with no persistent agent to install or maintain — faster to adopt for smaller or heterogeneous fleets, though it doesn't continuously self-correct drift the way a polling agent does unless you re-run it.

Example: `ansible-playbook -i inventory.ini deploy.yml` pushes a change to 50 servers over SSH immediately, whereas a Puppet-managed fleet would have each node's own agent pick up the equivalent change on its next poll cycle, commonly every 30 minutes, without anyone triggering anything centrally.

Question 32

What's the difference between a Kubernetes liveness probe and a readiness probe?

Accepted Answer

A liveness probe answers "is this container still alive, or should Kubernetes restart it?" — repeated failures cause kubelet to kill and restart the container, on the assumption it's in a broken, deadlocked state a restart can fix. A readiness probe answers a different question — "is this container currently ready to receive traffic?" — and if it fails, nothing gets restarted; the Pod is simply pulled from the Service's list of routable endpoints until the probe passes again, so no traffic is sent to a Pod that's temporarily busy warming a cache or reconnecting to a database.

Mixing them up causes real outages: relying on only a liveness probe for a slow-starting app (one loading a large model into memory, say) can get it repeatedly killed before it ever finishes starting, producing a crash loop that never completes.

The fix is a `startupProbe` for slow-starting apps (which suspends liveness/readiness checks until startup succeeds), a `readinessProbe` on a lightweight endpoint that only returns 200 once dependencies are actually ready, and a separate `livenessProbe` that just confirms the process hasn't deadlocked — so a temporarily-busy-but-alive Pod is pulled from traffic instead of being needlessly restarted.

Question 33

What is a Dockerfile, and what do the key instructions (FROM, RUN, COPY, ENTRYPOINT, CMD) do?

Accepted Answer

A Dockerfile is a text file with a sequence of instructions describing how to build a container image, layer by layer. `FROM` sets the base image everything else builds on top of. `RUN` executes a command at build time and commits its result as a new layer, like installing dependencies. `COPY` copies files from the build context (your local directory) into the image filesystem. `ENTRYPOINT` defines the fixed executable that always runs when a container starts, while `CMD` supplies default arguments to that entrypoint (or a default command if there's no ENTRYPOINT at all) that can be overridden at `docker run` time.

```dockerfile
FROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
ENTRYPOINT ["node"]
CMD ["server.js"]
```
Running `docker run myimage` executes `node server.js`, but `docker run myimage worker.js` overrides just the CMD portion, running `node worker.js` instead — ENTRYPOINT stays fixed as the interpreter, CMD is the swappable default argument.

Question 34

What's the difference between an SLA, an SLO, and an SLI?

Accepted Answer

An SLI (Service Level Indicator) is a directly measured metric of some aspect of service behavior — for example, the percentage of HTTP requests that complete in under 300ms, or the fraction of requests that don't return a 5xx status. An SLO (Service Level Objective) is an internal target set on top of that indicator — "99.9% of requests complete under 300ms, measured over a rolling 30-day window" — used internally to decide when reliability work should be prioritized over new features. An SLA (Service Level Agreement) is an external, often contractual promise to customers, usually a looser version of the SLO with real financial consequences (service credits, refunds) attached if it's breached — "99.9% uptime or a 10% bill credit that month."

The relationship matters: SLOs should be stricter than the corresponding SLA, giving engineering an early-warning buffer. If the internal SLO is 99.9% and the customer-facing SLA is 99.5%, the team gets to react and fix problems while still comfortably meeting the contractual promise, instead of finding out they're in breach only after the SLA itself has already been violated.

Question 35

What's the difference between a Kubernetes DaemonSet, a StatefulSet, and a Deployment?

Accepted Answer

A Deployment manages a set of interchangeable, stateless Pod replicas — any replica can be scaled up/down or replaced by an identical new one, and Pods have no stable identity since names and IPs change on every reschedule. A StatefulSet is for workloads needing stable, unique identity across restarts: each Pod gets a predictable, persistent name (`db-0`, `db-1`, `db-2`) and, if configured, its own dedicated PersistentVolume that follows it even when rescheduled to a different node — needed for something like a database cluster where each replica holds distinct data. A DaemonSet isn't about replica count at all — it ensures exactly one copy of a Pod runs on every node (or every node matching a selector), automatically adding a copy as new nodes join and removing it as nodes leave.

Example: a stateless API server uses a Deployment (any of its 5 identical replicas can be freely killed and replaced); a Kafka or Elasticsearch cluster uses a StatefulSet (each broker needs its own persistent identity and disk); a log-shipping agent like Fluentd uses a DaemonSet, since you want exactly one instance per node collecting that node's local logs, not a fixed count unrelated to cluster size.

Question 36

How would you debug a Kubernetes pod that's stuck in CrashLoopBackOff?

Accepted Answer

CrashLoopBackOff describes a restart pattern — the container keeps starting, exiting, and getting restarted with growing backoff delays — it isn't itself the root cause, so debugging means finding out why the process inside keeps exiting. Start with `kubectl logs --previous`, which shows logs from the crashed container instance rather than the freshly-restarted one that may have no logs yet — this usually surfaces the real error, like a missing env var or a failed database connection at startup. Then `kubectl describe pod ` and check the "Last State" section for the exit code and reason; `OOMKilled` with exit code 137 means the container hit its memory limit and was killed by the kernel, not a bug in application logic. Common root causes: a missing or misconfigured ConfigMap/Secret the app expects on startup, a dependency (database, message broker) that isn't reachable yet with no retry logic in the app, or a memory limit set too low for the app's real needs. Example: `kubectl describe pod myapp-7d9f` showing `Reason: OOMKilled, Exit Code: 137` immediately tells you to raise the memory limit or hunt for a leak, rather than treating it as a generic crash to debug purely from application logs.

Question 37

What are RPO and RTO, and how do they shape a disaster recovery strategy?

Accepted Answer

RPO (Recovery Point Objective) is how much data loss is acceptable, expressed as time — an RPO of 1 hour means after a disaster you can lose up to the last hour of data, so your backup or replication frequency must be at least that tight. RTO (Recovery Time Objective) is how long the system can stay down before it must be back up — an RTO of 4 hours means from the moment of failure, the system must be fully restored and serving traffic within 4 hours, which drives how much you invest in automation and standby infrastructure versus how much recovery can be manual.

These two numbers shape real architecture cost trade-offs: a payments system might need near-zero RPO (synchronous cross-region replication, expensive) and an RTO of minutes (an already-running warm standby, also expensive to keep idle), while an internal analytics dashboard might accept a 24-hour RPO (nightly backup) and a full day's RTO.

Example: a database with synchronous multi-AZ replication and automated failover gives near-zero RPO and an RTO of seconds to minutes, versus nightly `pg_dump` backups to S3 giving up to 24 hours of RPO and an RTO of however long a manual restore-and-verify process actually takes.

Question 38

What causes a Kubernetes pod to get OOMKilled, and how do you diagnose and fix it?

Accepted Answer

A Pod gets OOMKilled when a container inside it tries to use more memory than its `resources.limits.memory` setting — the kernel's cgroup memory controller, not Kubernetes itself, sends SIGKILL the instant that boundary is crossed, since cgroups enforce a hard ceiling rather than a graceful degradation. It shows up as exit code 137 (128 + signal 9), and `kubectl describe pod` reports `Reason: OOMKilled` under Last State.

To diagnose, check whether it's a genuine memory leak (usage climbs steadily over hours until it hits the limit and gets killed, then repeats — a sawtooth pattern on a memory-over-time graph) versus the limit simply being set too low for a legitimate peak workload, like a batch job that briefly needs more memory during one processing step.

Fix accordingly: raise the memory limit if the peak usage is legitimate and the node has headroom, add a profiler or heap-dump-on-OOM if it's a real leak, and for JVM-based apps specifically, make sure `-Xmx` is set explicitly below the container's memory limit — the JVM otherwise sizes its heap off the node's total memory rather than the container's limit, and can get OOMKilled even though "it never used that much" from the app's own view.

Question 39

What is GitOps, and how does it differ from a traditional push-based CD pipeline?

Accepted Answer

Traditional push-based CD has the pipeline itself push changes out to the target environment — a CI job runs `kubectl apply` or `helm upgrade` directly against the cluster, which means the CI system needs standing, live credentials with write access to production. GitOps flips this: a git repository is the single source of truth for the cluster's desired state, and an in-cluster agent (Argo CD, Flux) continuously watches that repo and pulls changes, reconciling the live cluster to match whatever's declared in git — the CI pipeline's only job becomes updating the git repo, not touching the cluster directly.

This matters for security, since no external system needs standing write credentials to production (the reconciler pulls from inside the cluster's own network boundary), and for drift correction: if someone manually edits something in production, Argo CD's continuous reconciliation loop notices the live state no longer matches git and alerts or auto-reverts it, whereas a push-based pipeline has no ongoing awareness of anything after its one-time push.

Example: an Argo CD `Application` resource pointed at a git path syncs the cluster to it every few minutes or on webhook, so a `git commit` bumping an image tag *is* the entire deployment action — no cluster credentials ever leave the CI runner.

Question 40

What is a service mesh, and what does it give you beyond native Kubernetes networking?

Accepted Answer

A service mesh is a dedicated infrastructure layer for service-to-service communication — mutual TLS encryption, retries, timeouts, circuit breaking, fine-grained traffic splitting, and detailed per-request telemetry — implemented by injecting a lightweight sidecar proxy (Envoy) alongside every application container, so every network call in and out of a Pod actually routes through that proxy instead of going directly between app containers. Kubernetes' native Service/kube-proxy layer gives you basic load balancing and service discovery, but nothing about encryption between services, nothing about retry/timeout policy, and only coarse traffic control — a mesh (Istio, Linkerd) adds all of that without application code having to implement retries, TLS, or tracing headers itself.

Example: with Istio you can declare "route 5% of traffic for service A to version v2" via a `VirtualService`, get automatic mTLS between every sidecar-injected Pod with zero app code changes, and get a uniform distributed trace plus a golden-signal dashboard (latency/traffic/errors/saturation) for every service in the mesh — capabilities that would otherwise require every team to individually implement retry logic, certificate management, and tracing instrumentation inside each service's own codebase.

Question 41

What is chaos engineering, and how would you design a "game day" exercise for a production system?

Accepted Answer

Chaos engineering is deliberately injecting controlled failures into a system — often in production or a production-like environment — to verify that the resilience mechanisms you believe exist actually work, rather than assuming they do because you wrote them once. A "game day" is a scheduled, planned exercise where a team deliberately triggers a specific failure scenario, like killing a database primary or injecting network latency between two services, and observes whether alerting fires, whether automated failover actually kicks in, and whether the on-call runbook holds up when followed under simulated pressure.

Designing one: pick a specific, bounded hypothesis — "if the primary Postgres instance dies, failover to a replica completes within 60 seconds with no data loss" — define a blast-radius limit and an abort condition (kill the experiment immediately if customer-facing error rate crosses a threshold), run it during low-traffic hours with the on-call team aware in advance (not a surprise, at least the first several times), and write down exactly what actually happened versus what was expected.

Tools like Chaos Mesh or AWS Fault Injection Simulator inject the specific failure — kill pod, inject latency, throttle CPU — in a controlled, reversible way inside Kubernetes rather than someone manually pulling a cable.

Question 42

What is the "error budget" concept in SRE, and how does it govern release velocity?

Accepted Answer

An error budget is the flip side of an SLO: if the SLO is 99.9% availability over 30 days, the error budget is the remaining 0.1% — a concrete, spendable allowance of acceptable failure, roughly 43 minutes of downtime a month. The idea, from Google's SRE practice, is to use that budget as an objective, pre-agreed governor of release velocity: as long as it isn't exhausted, the team can ship features and take reasonable risk freely; once it's burned through for the period, feature releases pause and effort shifts to reliability work until the budget recovers.

This turns a permanently tense negotiation — "can we ship this risky change or not?" — into a mechanical, data-driven decision instead of a political one; nobody has to argue in a meeting about whether the team's been "too risky lately," the burn-rate graph just answers it.

Example: a team notices their error budget burn rate spiked to 10x normal over the last 6 hours — a fast-burn alert — and by policy that automatically triggers a release freeze until the underlying issue is fixed and the burn rate returns to normal, regardless of how eager product is to ship the next feature.

Question 43

What does "toil" mean in the SRE sense, and why do SRE teams treat reducing it as a priority?

Accepted Answer

"Toil," in Google's SRE book sense, is work that is manual, repetitive, automatable, tactical, and produces no lasting value — it scales linearly with service growth rather than shrinking through improvement. Examples: manually restarting a service every time it hits a known memory leak, manually provisioning a new customer's environment by running the same 15 commands each time, or manually clearing a queue backlog every week. It's distinct from real engineering work, which produces a durable improvement — automation, a fix, a new capability — that permanently reduces future work.

SRE teams treat reducing toil as a priority (Google's own guidance caps toil at 50% of an SRE's time) because unchecked toil is a trap: it consumes exactly the time that would otherwise go toward the automation that would eliminate it, so a team buried in toil stays buried in it indefinitely, with its growth bounded on headcount rather than automation.

Example: instead of an engineer manually restarting a leaking service every night forever, the team spends a week writing an actual memory-leak fix or an automated restart-on-threshold policy — a one-time engineering cost that eliminates that toil source permanently.

Question 44

What is a Kubernetes PodDisruptionBudget, and when do you actually need one?

Accepted Answer

A PodDisruptionBudget (PDB) limits how many Pods of a given set can be voluntarily disrupted at once — it caps things like node drains during cluster upgrades, cluster-autoscaler scale-downs, or `kubectl drain`, but does NOT protect against involuntary disruptions like a node crashing unexpectedly. You declare either `minAvailable` (at least N or N% of Pods must stay up) or `maxUnavailable` (at most N or N% can be down at once), and Kubernetes' eviction API refuses a voluntary eviction that would violate the budget, forcing the operation to wait until it's safe.

You need one for any service where losing too many replicas at once during routine maintenance would cause an outage — a 3-replica database quorum, or a service with strict availability requirements.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: api }
```
With this in place, draining a node hosting 2 of the api's 3 replicas simultaneously blocks on the second eviction — since only 1 would remain, violating `minAvailable: 2` — until a replacement Pod schedules elsewhere and becomes ready. Without the PDB, both could be evicted at once, potentially dropping the service below needed capacity during ordinary maintenance.

Question 45

Explain etcd's role in a Kubernetes cluster and what happens if the etcd quorum is lost.

Accepted Answer

etcd is the distributed key-value store that serves as Kubernetes' entire source of truth — every object (Pods, Deployments, Services, ConfigMaps, everything `kubectl get` can show you) is stored there, and the API server reads and writes to etcd for essentially every operation; nothing else in the control plane keeps its own independent copy of cluster state. etcd achieves consistency via the Raft consensus algorithm, which requires a strict majority (quorum) of its member nodes to agree before any write commits — a 5-node etcd cluster tolerates 2 node failures and keeps working (3 remain, still a majority), but if 3 of 5 go down, only 2 remain, which is not a majority.

When quorum is lost, etcd cannot commit any new writes — reads of already-committed data may still work in some configurations, but the cluster becomes effectively read-only or fully unavailable: no new Pods can be scheduled, no Deployment can be updated, nothing can change state, even though existing running Pods keep running independently (kubelet manages containers it already knows about on its own).

Recovering from a full quorum loss requires restoring from a backup snapshot (`etcdctl snapshot restore`), which is why etcd backups are one of the most critical, non-optional pieces of Kubernetes operations, and why production clusters run etcd across 3 or 5 nodes spread over separate availability zones rather than fewer.

Question 46

What is "cardinality explosion" in a Prometheus-based metrics system, and how do you prevent it?

Accepted Answer

Cardinality of a metric is the number of unique time series it produces, driven by every combination of its label values. Cardinality explosion happens when a label is given an unbounded or high-variance value — putting a raw user ID, a full request path with dynamic IDs embedded, or a UUID directly into a label — because Prometheus creates a brand-new time series for every unique label combination, and each series has real memory, storage, and query cost. A metric like `http_requests_total{user_id="12345", ...}` on a service with a million users effectively creates a million separate time series from one metric definition, which can exhaust Prometheus's memory and make queries catastrophically slow, sometimes taking down the whole monitoring stack.

Prevention: never put high-cardinality values (user IDs, session IDs, raw request paths, timestamps, IP addresses) into metric labels — put them into log lines or trace spans instead, which are built for high-cardinality data — and keep metric labels to a small, bounded set (`method`, `status_code`, a templated `route` like `/users/:id` rather than `/users/12345`).

Example: `http_requests_total{method="GET", route="/users/:id", status="200"}` stays bounded regardless of how many actual users hit that route, because `:id` is templated rather than the raw dynamic value.

Question 47

What is the "noisy neighbor" problem in a multi-tenant Kubernetes cluster, and how do resource requests/limits and quotas mitigate it?

Accepted Answer

A "noisy neighbor" is a workload on a shared multi-tenant node (or cluster) that consumes a disproportionate share of a shared resource — CPU, memory, disk I/O, network bandwidth — degrading performance for other, unrelated workloads scheduled on the same node, even though those workloads did nothing wrong themselves. It happens when Pods don't have resource requests/limits set, or have them set too loosely, so the scheduler can't reason about real capacity and nothing at the kernel/cgroup layer forces a greedy Pod to stay within its fair share.

Mitigation: set `resources.requests` (used by the scheduler for placement, guaranteeing that much is reserved) and `resources.limits` (a hard ceiling enforced by the kernel's cgroup — CPU gets throttled, memory over the limit gets OOMKilled) on every Pod, so no single workload can silently starve its neighbors.

For stronger isolation, use a `ResourceQuota` at the namespace level to cap total consumption per team/tenant, and a `LimitRange` to force every Pod in a namespace to declare requests/limits, rejecting ones that don't. For the most demanding or most sensitive tenants, dedicate separate node pools (with taints/tolerations) so their workloads are never even scheduled onto the same physical node as anyone else's.

Question 48

What is a "split-brain" scenario in a distributed consensus system like etcd or ZooKeeper, and how does quorum-based consensus prevent it?

Accepted Answer

Split-brain is a failure mode where a network partition causes two (or more) subsets of nodes to each believe they're the sole authoritative leader and independently accept writes — since neither side can see the other, both proceed as if they're in charge, and once the partition heals you're left with two divergent, conflicting histories of the data that have to be reconciled, often with real data loss since one side's writes must be discarded or manually merged. It's especially dangerous in systems relying on simple heartbeat-based leader election rather than a formal consensus protocol, where a brief network blip can cause both sides to elect their own leader at once.

Quorum-based consensus (Raft in etcd, ZAB in ZooKeeper) prevents this by requiring a strict majority of nodes to agree before any decision — including "who is the leader" — is valid. In a 5-node cluster split by a network failure into a 3-node partition and a 2-node partition, only the 3-node side has a majority and can elect a leader or accept writes; the 2-node side, lacking a majority, refuses to elect its own leader or accept writes at all — so there's never a moment where two sides both believe they're authoritative simultaneously.

This is exactly why etcd/ZooKeeper clusters are always deployed with an odd number of nodes: an odd total guarantees a partition into two groups can never produce two majorities at once.

Question 49

How would you automate zero-downtime TLS certificate rotation for mTLS between services in a service mesh?

Accepted Answer

In a service mesh doing mTLS, every service holds a short-lived TLS certificate issued by an internal CA, used both to prove its own identity and verify the identity of services it talks to — manually rotating these across hundreds of services on a schedule is impractical by hand. The standard approach pairs cert-manager (in Kubernetes) with the mesh's own certificate authority (Istio's Citadel/istiod, or a mesh-integrated Vault PKI backend): certs are issued with a short TTL, often 24 hours or less, and a controller continuously watches expiry and reissues well before it, pushing the new cert into the sidecar proxy's in-memory store without restarting the workload container.

Zero downtime comes from the sidecar proxy (Envoy) supporting hot-reload of its TLS certificate via its Secret Discovery Service (SDS) API — the control plane pushes the new cert/key, the proxy starts presenting it on new handshakes immediately, while already-established connections keep using whichever cert was current when they were negotiated, so nothing gets dropped or restarted.

This is precisely why meshes deliberately favor very short cert lifetimes over the year-long certs common for public-facing TLS: short lifetimes make the automated rotation machinery load-bearing and routine rather than a rare, risky manual event, and limit the blast radius if a cert or key is ever compromised.

Question 50

What is the "thundering herd" problem in autoscaling, and how do you design around it?

Accepted Answer

The "thundering herd" problem in autoscaling happens when a sudden traffic spike causes many new instances or Pods to be provisioned simultaneously, and all of them hit some shared downstream resource — a database, a cache, a cold dependency — at the same moment during their own startup routine, like each one independently warming a local cache from the same database query or opening a fixed pool of connections at once. The very scale-out meant to absorb the load spike instead creates a secondary spike against a shared bottleneck that wasn't sized for that many simultaneous startups, sometimes taking down the database that was handling the original traffic just fine.

Mitigations: stagger new instances' startup work with random jitter so they don't all fire at the same instant, use connection pooling with sane per-instance limits so N new instances don't each open M connections and multiply against the database's `max_connections`, warm caches from an already-populated shared source (like Redis) rather than each instance querying the primary database cold, and scale out incrementally rather than jumping straight from 3 to 50 instances in one step.

Example: an autoscaling group scaling from 5 to 60 instances, each configured to add `random(0, 30)` seconds of jitter before its first database connection attempt, spreads what would have been 55 simultaneous new connections across a 30-second window instead of a single instant.

DevOps Interview Questions & Answers

Ready to practice these live?