A few days ago I got pulled into a curious conversation: the team was excited, implementing autoscaling on queue-consumer pods, trying to increase throughput.
All good so far, right?
But then came the question:
“Why are we spinning up more pods if we’re not even using the resources well inside a single pod?”
Silence.
And that’s exactly where the problem lives. Just because your infrastructure is scalable doesn’t mean you should go full “Let it Go” on the pods — and leave the bill (of CPU, memory, and chaos) for someone else to deal with.
Today, let’s talk about a common mistake: confusing concurrency with parallelism.
Spoiler: they’re not the same thing. And the difference between them can decide whether your system scales well or just burns resources.
Concurrency ≠ Parallelism
Concurrency
Concurrency is about dealing with multiple tasks at the same time, but not necessarily executing them simultaneously.
It’s as if a single core were handling several things at once, jumping from one to another whenever it hits a wait (like I/O calls, database access, external APIs, etc).
You gain performance not by doing everything at the same time, but by not sitting idle waiting.
Practical example:
- A worker consumes a queue, makes a request to an API and… waits.
- During that time, it could be processing another task.
- Solution: apply concurrency, like goroutines (Go), lightweight threads (Java), async/await (Node, Python).
Parallelism
Parallelism, on the other hand, is when you have multiple cores processing different things at the same time, for real.
Ideal for heavy tasks, like data transformation, batch processing, machine learning, etc.
Practical example:
- Processing 1 million records with Spark, distributing the work across multiple nodes at the same time.
But be careful: blind parallelism with heavy I/O can be inefficient.
You spin up a bunch of workers… that just sit there waiting for a network response.
When to use each one?
Use concurrency when…
- Your code depends on external calls (APIs, databases, legacy systems)
- You want better utilization of a single resource (e.g. a pod)
Use parallelism when…
- You’re dealing with CPU-bound tasks
- You need to scale real processing, not just orchestration
Even though I explain them separately, using one doesn’t rule out the other! In fact, you need to use both — but the way you implement each is different.
Think of it like this: inside a single pod, you use concurrency so you don’t waste time sitting in waits. Once that pod is already at a healthy limit, then you scale horizontally (real parallelism, more replicas). Order matters — scaling before optimizing means paying a premium for inefficiency.
The SRE connection
This is where it stops being operating-system theory and becomes a reliability and cost decision.
- Eliminating toil and waste: spinning up a pod by reflex is the opposite of efficiency. Before touching the HPA, look at the real utilization inside the pod. If a worker spends 80% of its time waiting on I/O, the problem isn’t a lack of replicas — it’s a lack of concurrency.
- Monitoring that matters: CPU and memory aren’t enough. Measure effective per-pod utilization, queue depth, processing latency, and time spent waiting on I/O. Those signals tell you which lever to pull.
- Error budget and blast radius: uncontrolled concurrency turns into race conditions and deadlocks in production — and that burns error budget fast. Parallelism without rate limiting takes down downstream dependencies. Reliability is about sizing with intention.
- FinOps is an SRE responsibility: every unnecessary replica is money burned every single month. Making good use of a single pod before scaling is cost optimization at its core.
And the risks?
Concurrency without control
- Pushing too much concurrency makes it constantly switch context between tasks (context switching), and excessive context switching generates overhead.
- It can blow up into race conditions, deadlocks, and freeze everything (which is why you need to be careful and handle it with mutexes and locks).
Poorly sized parallelism
- Creating too many pods without evaluating real usage.
- Increasing the number of threads without control (and hardware is finite, remember?).
- Scaling without applying rate limits → a disaster waiting to happen.
Conclusion
The difference between concurrency and parallelism may seem subtle, but ignoring it can be costly — in performance, in money, and in unnecessary complexity.
Go back to the beginning: the team wanted more pods. But the right question wasn’t “how many replicas?” — it was “am I making good use of what I already have?”.
The truth is that a lot of architecture is being decided on impulse. Solutions get sold as indispensable, the hype takes over… and suddenly we’re paying a premium for infrastructure that doesn’t even bring real value to the business.
A good technical choice is one made with intention, clarity, and purpose — understanding what your system actually needs before you start scaling.
If this kind of reflection resonates with you, and you want to learn how to use technology to scale your technical capacity (without becoming a hostage to tooling), make sure to follow me on social media and keep up with the newsletter!
Cheers,
Douglas Mugnos
MUGNOS-IT 🚀
