kubectl apply -f deployment.yaml

But have you ever wondered…

What actually happens inside Kubernetes after you hit Enter?

You are basically telling Kubernetes:

“Hey, I want my app to run like THIS (as written in the file). Make it happen.”

Super Simple Analogy (Pizza Shop)

Think of Kubernetes like a smart pizza shop OR pizza restaurant:

• You = Customer (kubectl)

• deployment.yaml = Your order (YAML file)

• Kubernetes = Restaurant system

• Pods = Chefs making pizza

You’re basically saying:

“I want my app to run like THIS. Make it happen.”

Step-by-step in real life terms what actually happens behind the scenes inside Kubernetes?

Let’s break it down in a simple way 👇:

1️⃣ You send a request

kubectl talks to the API Server 📡
kubectl is just a client

You run the command.

• kubectl reads the YAML file → converts it into JSON → sends request via REST API over HTTP

The kubectl command sends your request (create Pod, deploy app) to the Kubernetes API Server. kubectl does not deploy anything itself

Analogy: You place your order at the counter

• kubectl = You speaking

• API Server = Reception desk

“I want 2 pizzas like this!”

2️⃣ Kubernetes checks everything

API Server is the entry point

API Server validates & processes the request

All requests go through kube-apiserver
The API Server checks the YAML file and verifies if the configuration is valid.

Central communication hub

API server is the “front door” of the cluster.

What actually happens:

Before storing in etcd, Kubernetes also runs:

• Who are you? (authentication)

• Are you allowed? (authorization)

• Is your request valid? (admission checks) (VERY important)

If everything is fine → it accepts your request.

These can:

• modify your request (e.g., inject sidecars like Istio)

• reject it (policy enforcement)

Before persisting the request, the API server runs it through an internal request pipeline including authentication, authorization, and admission controllers.

Before saving, Kubernetes may also modify or enforce rules using admission controllers.

Analogy: Reception checks your order

• Is it readable?

• Is it valid?

If wrong → rejected.

If correct → accepted

3️⃣ It saves your request

The desired state of resources is stored in etcd as Kubernetes objects.
Once validated, the desired/cluster state (source of truth) stores your Deployment in etcd (the Kubernetes database).

👉 Important:
At this point:

• Your app is NOT running

• No containers exist yet

It’s just a saved instruction saying:
“Hey, I want 3 copies of this app.”

Analogy: Order is saved in system database

• etcd = Restaurant’s order register

Important:
This is the “desired state”
(what you WANT)

4️⃣ Controllers create pods

Controller Manager detects changes (automatically)

Now the smart part.

Controllers continuously watch the API server for changes and ensure the desired state matches the actual state. They are always watching.

Creates/updates resources

Different controllers exist:

• Deployment Controller sees your request → creates a ReplicaSet

• ReplicaSet Controller sees that → creates 3 Pods

• Node Controller

At this stage, Pods are created as Kubernetes objects, but they are not yet scheduled or running on any node.

Different controllers (like Deployment controller) watch the state and act accordingly.

Deployment → ReplicaSet → Pod

Analogy: Manager checks if order is being prepared

• “Customer wants 2 pizzas”

• “Do we have 2 pizzas ready?”

If not → it tells staff:
“Start making!”

5️⃣ Scheduler picks machines

Scheduler assigns the Pod to a Node

Scheduler updates the Pod’s nodeName field
The scheduler decides which worker node should run the pod.

👉 “Which machine should run each pod?”

It picks the best nodes and updates the info.

Still no containers running yet — just assignments.

Analogy: Decides which chef will cook

• Which kitchen station is free?

• Which has resources?

Assigns work

6️⃣ Kubelet creates the container

Kubelet runs the containers
On each selected machine, the kubelet:

• The kubelet on that node pulls the image,

• starts the container and

• Reports status.

Kubernetes uses container runtime (like containerd, not always Docker)

Now the real action.

Each machine has a kubelet. It sees:

“Oh, this pod is assigned to me.”

So it:

• Downloads the container image

• uses CNI (Container Network Interface) for networking and CSI (Container Storage Interface) for storage

• Starts the container

Now your app is finally running!

Analogy: Actual chef starts cooking

• Pulls ingredients (container image)

• Starts making pizza (container)

Key idea (very important)

• Everything goes through the API server

• Most control-plane components communicate via the API server, ensuring loose coupling. i.e Some components do have direct communication paths (e.g., kubelet → container runtime)

• Each one just watches and does its job

One-line takeaway

kubectl apply doesn’t run your app — it just tells Kubernetes what you want, and Kubernetes makes it happen step by step.

Flow: Event-driven nature (not linear)

Who’s involved (in simple terms)

Think of Kubernetes like a system with different workers:

• kubectl → Just a messenger. It sends your request.

• API server → The main gatekeeper. Everything goes through here.

• etcd → The database. It stores everything.

• Controllers → The “brains” that figure out what needs to be created.

• Scheduler → Picks which machine should run your app.

• Kubelet → The worker on each machine that actually runs containers.

Reality:

• It’s event-driven + continuous

• Controllers are always watching

• Not a one-time chain

Most components communicate via the API server, ensuring loose coupling.

The big idea (most important part)

Here’s the surprising truth:

• The Controller Manager does NOT directly talk to Kubelet

• Most Kubernetes components interact through the API server, though some internal components (like kubelet and container runtime) communicate directly.

• Everything goes through the API server.

  • Watches cluster state via the API server (which reads from etcd)

  • Sends updates back to API Server - Everyone writes back to it

It’s like a shared notice board.

• Then:

  • Scheduler assigns pods

  • Kubelet reads instructions from API Server

👉 So the API Server is the central hub — everything connects through it.

This is not a one-time flow — Kubernetes continuously watches and reacts.

But the REAL truth:

• Nothing runs instantly

• Nothing runs directly

• Everything goes through API Server

• Everything is watched continuously

You declare → Kubernetes stores → Controllers notice → System fixes → Repeat forever

Kubernetes is NOT executing commands
It is continuously correcting reality to match your YAML

Easy analogy (post office)

Think of it like sending a package:

1. You submit a request (kubectl)

2. Post office accepts it (API server)

3. It gets recorded (etcd)

4. Workers prepare packages (controllers)

5. Planner assigns delivery trucks (scheduler)

6. Drivers deliver it (kubelet runs containers)

Components don’t directly call each other — they just check the system and do their job.

Key takeaway

• kubectl apply doesn’t run anything

• It just declares what you want

• Kubernetes slowly makes it real, step by step

Final Result:

✔ Pod gets created
✔ Container starts running
✔ Application becomes available
✔ Deployment successful

DevOps Insight:

Kubernetes always works on the concept of:
Desired State vs Actual State

Desired State = What you wrote in YAML
Actual State = What is really running

Kubernetes constantly checks:

“Are these equal?”

If not:
👉 It automatically fixes it

Kubernetes doesn’t care what you ran.
It only cares what you WANT.

Example (super simple)

You said:

“Run 3 copies of my app”

But 1 crashes

Kubernetes says:
“You wanted 3… I see only 2… let me create 1 more.”

That’s the power.

One-line summary

👉 kubectl apply is not just a command —
it starts a chain reaction inside Kubernetes that makes your app run automatically.

Controllers continuously work to make the cluster match the desired configuration.

If you're working with Kubernetes daily, this simple command actually triggers multiple components inside the cluster

kubectl apply in 10 seconds

You don’t run containers.
You declare a state.
Kubernetes makes it true.

“Now that we understand the basic flow, let’s go one level deeper into what actually happens behind the scenes"

1. Apply Strategy (3-Way Merge)

One critical detail often overlooked is how kubectl apply actually updates resources.

It uses a 3-way merge strategy involving:

It uses a 3-way merge strategy involving:

• Last applied configuration
(stored as last-applied-configuration annotation for client-side apply, or managed fields for server-side apply)

• Current live state in the cluster

• New configuration from your YAML

This allows Kubernetes to:

• Apply only the changes you’ve made • Avoid overwriting fields managed by other controllers or users

2. Admission Controllers

Before your request is persisted, it passes through admission controllers, which are critical in real-world clusters.

There are two main types:

Mutating Admission Controllers Modify requests (e.g., injecting sidecars like service meshes) Validating Admission Controllers Enforce policies and can reject requests

These controllers can:

Automatically alter your resource definitions Completely block deployments

In production, this layer is often where security and compliance policies are enforced.

Examples:

• PodSecurity

• ResourceQuota

• LimitRanger

3. Watches & Informers Are Not Emphasized

The system is not a direct chain of calls between components.

Instead:

Components do not call each other directly They communicate via: Watches (event streams from the API server) Shared state (stored in etcd)

For example:

Controllers watch for changes instead of being “triggered”

This is a fundamental architectural concept that explains how Kubernetes remains loosely coupled and scalable.

4. Not a Linear Flow — It’s Event-Driven

The process is often described as a step-by-step pipeline, but in reality:

Kubernetes is event-driven and asynchronous

This entire Controllers process run by reconciliation loops that continuously push the system toward the desired state.

This means:

State changes propagate over time, not instantly Multiple components act independently

This explains real-world behaviors like delays, retries, and eventual consistency.

5. API Semantics (PATCH vs POST vs PUT)

kubectl apply is not always creating resources.

kubectl apply typically sends a PATCH request, but may use POST for creation if the resource does not exist.

This enables:

Incremental updates Field-level changes instead of full replacements

👉 Understanding this is key for debugging conflicts and API behavior.

6. Missing Real-World Failure Scenarios

The article focuses on the “happy path,” but real systems often fail in different ways:

Common issues include:

• ImagePullBackOff – container image cannot be fetched • Pending Pods – scheduler cannot find a suitable node • CrashLoopBackOff – application keeps failing • Resource constraints – insufficient CPU/memory

👉 These scenarios are essential for troubleshooting and operating Kubernetes in production.

💡 Final Thought

The high-level flow is correct, but Kubernetes is far more nuanced under the hood. Understanding these deeper concepts is what separates basic familiarity from production-ready expertise.

There can be slight delays between each step due to the asynchronous and distributed nature of Kubernetes.

Kubernetes is eventually consistent, not instantly consistent.

You can observe this flow using:

• kubectl describe pod

• kubectl get events

Sometimes it's just configuration (rules), not a real error.

What confused me?

While debugging a Pod, I saw this:

• Liveness: tcp-socket :80 delay=15s timeout=1s period=20s #success=1 #failure=3

• Readiness: tcp-socket :80 delay=5s timeout=1s period=10s #success=1 #failure=3

At first glance, it looked scary.

#success=1 #failure=3

It looks like(Failure History):
“Oh no, the Pod already failed 3 times!”

But that’s wrong.

It actually means (Threshold Rule):
“Fail 3 times in a row BEFORE taking action”

Kubernetes doesn’t react to single failures — it evaluates patterns over time using defined thresholds.

Those numbers are rules, not history.

What is tcp-socket :80 in Kubernetes?

tcp-socket :80

It means:

Kubernetes is trying to open a TCP connection to port 80 on the container.

Simple meaning

• TCP → basic network connection (like “can I reach you?”)

• port 80 → where your app is expected to listen (usually HTTP)

So Kubernetes is asking:

“Can I connect to this container on port 80?”

What each field means (quick decode):

delay=15s → wait before first check 
timeout=1s → max time for probe response 
period=20s → check every 20 seconds 
#failure=3 → fail 3 times → action 
#success=1 → 1 success → mark healthy

⚙️ What are Liveness & Readiness?

🔹 Liveness Probe

• Checks: “Is the container alive?”

• If it fails 3 times continuously → Kubernetes restarts the container

Like checking if an app is frozen and restarting it.

🔹 Readiness Probe

• Checks: “Is the container ready to serve traffic?”

• If it fails 3 times continuously → Kubernetes stops sending traffic to it

Like removing a sick player from a match.

• Liveness = “Should I restart?”

• Readiness = “Should I send traffic?”

The Important Detail

#failure=3 means:

NOT: “failed 3 times already”

YES: “will tolerate up to 3 failures before acting”

When does Kubernetes actually take action?

Understanding the threshold is not enough — timing also matters.

Kubernetes waits based on this formula:

Action Time = failureThreshold × periodSeconds

Example

failureThreshold = 3  
periodSeconds = 10

Kubernetes waits 30 seconds before taking action

What happens during that time?

• 1st check → fail
  (wait 10 secs)

• 2nd check → fail
  (wait 10 secs)

• 3rd check → fail
  ➡️ Now Kubernetes acts

Kubernetes is a control system — it continuously evaluates state and acts only when thresholds are crossed.

What action does Kubernetes take?

Liveness → restart the container

Readiness → stop sending traffic

Why this matters

• Your app might fail briefly and still be fine

• Kubernetes gives it time before reacting

This is part of its fault tolerance design

If you misunderstand this, you might:

• Panic unnecessarily

• Debug the wrong thing

• Waste time chasing “fake errors”

Where you SHOULD actually check

Instead of focusing on that line alone:

1. Check events
kubectl describe pod

Look for: "Unhealthy"

2. Check pod status

kubectl get pods

Look at: READY column

3. Check service issues

If a service has no endpoints:

• Readiness probe may be failing

• OR labels don’t match

Big Lesson (Very Important)

Kubernetes doesn’t react instantly — it waits and tolerates some failure.

This is called:

➡️ Fault tolerance

Meaning:

• One failure = OK

• Two failures = still OK

• Three failures in a row = now action

Simple Analogy

Think of it like alarms:

• You don’t call the fire brigade for 1 spark

• You wait until it’s clearly a fire

Kubernetes does the same.

One-Line Takeaway

Those numbers are rules, not history — they define when Kubernetes acts, not what already happened.

Debug real failures, not configuration.

Final Thought

Kubernetes is deliberately patient, not slow.
That patience is what gives you fault tolerance.

Kubernetes delays action intentionally — it only acts after the failure threshold is exceeded.
Understanding this saves a lot of debugging time.

Real-World Examples (Simple & Practical)

Here’s how these concepts show up in real production issues:

1. App/Pod Keeps Restarting (Liveness Probe)

What you see:
Your Pod keeps restarting again and again.

What’s happening:

• Your app gets stuck (maybe infinite loop, deadlock, or crash)

• Liveness probe checks: “Are you alive?”

• It fails 3 times consecutively → Kubernetes restarts the container

Like force-closing a frozen mobile app

Important:
This is NOT Kubernetes failing

It’s Kubernetes trying to fix your app automatically

2. App Running But No Traffic (Readiness Probe)

What you see:

• Pod is running

• But users can’t access your app

What’s happening:

• Readiness probe checks: “Are you ready?”

• Your app is slow / still starting / DB not connected

• It fails 3 times consecutively → Kubernetes stops sending traffic

Like a restaurant that is open but not ready to serve customers yet

3. CrashLoopBackOff (Common Confusion)

What you see:

CrashLoopBackOff

Looks scary

What’s actually happening:

• Your app crashes on startup (bad config, wrong env variables, etc.)

• Liveness keeps restarting it

• It fails again → restart → fail → restart…

1. Loop continues → CrashLoopBackOff

   Kubernetes is NOT the problem
   Your app/config is the problem

4. Service Not Working (No Endpoints)

What you see:

• Service exists

• But no traffic reaches your Pod

What’s happening:

• Readiness probe is failing

  OR

• Labels don’t match

1. Kubernetes removes your Pod from traffic

2. Like a delivery app hiding a restaurant that isn’t ready

Simple Summary

Real-world mapping:

- Readiness failing → No traffic
- Liveness failing → Restarts / CrashLoopBackOff
- Service mismatch → No endpoints

Final Insight

• Kubernetes is not failing randomly

• It is protecting your system using rules

Once you understand this, debugging becomes much easier:

• You stop panicking

• You debug faster

• You fix the real problem

Real Production Troubleshooting Checklist

Step-by-Step Kubernetes Debugging Checklist

1. Check Pod Status

kubectl get pods

Look for:

• Restarting?

• Not ready?

• CrashLoopBackOff?

2. If Restarting → Check Liveness

Ask:

• Is app crashing?

• Infinite loop?

• Memory/CPU issue?

Fix:

• Logs

• App bug

• Config issue

3. If Running But Not Working → Check Readiness

Ask:

• App still starting?

• DB/API not connected?

• Health endpoint failing?

Fix:

• Increase delay

• Fix dependencies

• Check /health endpoint

4. If CrashLoopBackOff

Most common causes:

• Wrong environment variables

• Missing secrets

• Bad config

Rule:
Kubernetes is restarting → your app is failing

5. If No Traffic

Ask:

• Service connected?

• Labels correct?

👉 Check:

kubectl get svc
kubectl get endpoints

6. Always Check Logs

kubectl logs <pod-name>

This tells the real story

7. Golden Rule

Don’t panic at config output

Instead ask:

• Is it restarting?

• Is it serving traffic?

• Is it actually broken?

Summary:

If someone asks: “How do you debug a failing Pod?”

You can say:

1. Check Pod status

2. If restarting → Liveness issue

3. If no traffic → Readiness issue

4. If crashing (CrashLoopBackOff) → logs/config issue

5. If no endpoints → service/labels mismatch

kubectl get pods

kubectl describe pod

kubectl logs

kubectl get svc

kubectl get endpoints

One-Line Power Statement

“Kubernetes doesn’t fail randomly — it enforces rules with patience. Your job is to understand which rule is being triggered.”

Next time you see a failure in Kubernetes,

don’t ask: “Why did it fail?”

Ask: “Which rule is being triggered?”

Kubernetes architecture is actually very straightforward once you split it into two parts –
1. Control Plane
2. Worker Nodes
Together they form the Cluster.

CONTROL PLANE — This is the brain of the cluster

1. API Server

This is the central communication point (Heart of the Cluster) of Kubernetes.
Everything — kubectl commands, controllers, kubelets — talks to the API Server.

It:

• Receives and validates every request

• Talks to etcd to read/update cluster data

• Ensures everyone communicates in a standard format

So if I run:

kubectl get pods

the API Server handles that entire request.

2. etcd

This is the database of Kubernetes.
It stores the entire cluster state, including:

• Pods

• Deployments

• Services

• Secrets / ConfigMaps

• Node information

Anything the API Server needs, it reads or writes from etcd.

3. Scheduler

The Scheduler looks for Pods that are not yet assigned to a Node.

It decides:

• Which Pod should run on which Node

• Based on CPU, memory, taints, affinity, etc.

Once it picks a node, it updates the API Server so the plan is recorded in etcd.

4. Controller Manager

Controller Manager ensures that the desired state = actual state.
It continuously watches the cluster.

Example:

• If a Pod crashes, the Controller Manager will recreate it automatically.

• If a node dies, it reschedules pods somewhere else.

• It basically keeps the cluster healthy and stable.

5. Cloud Controller Manager

This component talks to cloud provider APIs
(AWS, Azure, GCP, DigitalOcean, etc.)

It manages:

• Load balancers

• Volumes (EBS, Persistent Disks)

• Node lifecycle (add/remove nodes)

It is what lets Kubernetes work smoothly in the cloud.

WORKER NODE — This is where applications run

Each Worker Node hosts Pods, and has three main components.

1. Kubelet

This is the node agent.

It:

• Talks to API Server

• Makes sure containers are running as expected

• Reports issues back to Control Plane

If a container fails, kubelet will restart or report it.

2. Container Runtime Interface (CRI)

This is the engine that actually runs containers.

Examples:

• Docker

• containerd

• CRI-O

Kubelet uses this runtime to create and manage containers.

3. kube-proxy

This handles networking inside the node.

It:

• Manages forwarding rules

• Ensures Pod-to-Pod communication

• Enables access to Services

Without kube-proxy, nothing in the cluster could talk to each other.

4. Pods

These are the smallest deployable units in Kubernetes.
A Pod may contain:

• 1 container

• or multiple tightly coupled containers

Applications actually run inside Pods.

Overall, Kubernetes works because the Control Plane makes decisions and maintains cluster state,
and the Worker Nodes focus on running workloads.
This separation is what gives Kubernetes reliability, self-healing and scalability.

Questions using SAME ANSWER:

1. Explain Kubernetes architecture and its components.

2. Explain Control Plane vs Data Plane architecture.

3. What components run on the Control Plane?

4. What components run on Worker Nodes?

5. Explain EKS architecture (because EKS = same K8s architecture + managed control plane).

➡️ One architecture answer covers all these.

Core Answer:
Control Plane components (API server, etcd, scheduler, controller-manager) +
Worker Node components (kubelet, kube-proxy, container runtime).

Explain Kubernetes architecture in a simple way in interview:

Kubernetes mainly has two big parts — the Control Plane, which is like the brain, and the Worker Nodes, where your actual applications run. Together, all of this forms the Kubernetes cluster.

Starting with the Control Plane:

The most important component is the API Server. Think of it as the heart or the communication center. Every request — whether it’s coming from kubectl, from another component, or even from automation tools — goes through the API Server. It talks to etcd, which is the cluster’s database. etcd stores everything about the cluster: pods, secrets, configmaps, services, deployments — basically the entire state.

Then we have the Scheduler. Its job is simple: whenever a new Pod is created, the Scheduler decides which worker node it should run on. The decision is based on resource availability like CPU, memory, taints, affinity, etc.

Next is the Controller Manager. This component continuously watches the cluster’s desired state versus the actual state. If something drifts — like a Pod crashes — the controller manager steps in to fix it. It ensures the cluster maintains the state defined in our manifests.

Now, on the Worker Node side:

Each node has a kubelet, which is the main agent. kubelet ensures that the containers described by the API Server actually run on the node. If something is wrong, kubelet reports it back.

The next part is the Container Runtime, like Docker, containerd, or CRI-O — this is the software that actually runs our containers.

We also have kube-proxy, which handles networking on the node. It makes sure Pods can talk to each other and to Services by managing the necessary routing rules.

Finally, the Node runs Pods, which are the smallest deployable units in Kubernetes. A Pod may contain one or more containers that always run together on the same node.

In summary, Kubernetes provides a very resilient and self-healing architecture. The Control Plane decides what should happen, and the Worker Nodes execute how it should happen. Because of this separation, Kubernetes can auto-restart Pods, scale applications, and roll out updates smoothly — even when things fail.