Concurrency Limiter Rate Limiting Algorithm

Introduction

Concurrency Limiter is a rate limiting algorithm that focuses on active requests. Unlike algorithms such as Token Bucket, Fixed Window, or Sliding Window, it does not mainly ask, “How many requests came in this time period?”

Instead, it asks a different question: “How many requests are running right now?”

This makes it useful when the main problem is limited server capacity, database connections, slow downstream services, or expensive computations.

What Is a Concurrency Limiter?

A Concurrency Limiter restricts the number of requests that can execute at the same time. It does this by maintaining a fixed number of available slots.

Each slot represents permission for one request to run. When a request starts, it occupies a slot. When the request finishes, the slot is released and becomes available again.

For example, if a server can safely handle 1,000 simultaneous requests, the system may configure only 850 active slots. This leaves some extra capacity for stability instead of pushing the server to its absolute limit.

How Concurrency Limiter Works

The working is based on slot allocation and release.

Request arrives => Slot is checked => Request starts or waits/rejects => Slot is released after completion

The basic steps are:

• Define slots: The system decides the maximum number of concurrent requests allowed.

• Check availability: Every new request checks whether a slot is free.

• Occupy slot: If a slot is available, the request starts processing.

• Hold slot: The request keeps the slot for its entire execution time.

• Release slot: After processing completes, the slot becomes available again.

• Handle overflow: If no slot is free, the request is either rejected or placed in a queue.

This keeps the number of active requests bounded at all times.

What Happens When All Slots Are Full?

When every slot is already occupied, the system has two common choices.

• Immediate rejection: The request is rejected quickly because the system has no safe capacity left.

• Waiting queue: The request waits until another request finishes and releases a slot.

Immediate rejection is useful when waiting would make the user experience worse or when the system wants to fail fast. A waiting queue is useful when short delays are acceptable and the system wants to avoid rejecting requests too aggressively.

If HTTP APIs reject excess requests, they may return a status such as: HTTP 429 Too Many Requests

Some systems may also return service-unavailable style responses depending on where the limiter is implemented.

Where Concurrency Limiter Is Useful

Concurrency Limiter works well when system resources are limited and active workload matters more than request frequency.

Common use cases include:

• Database connections: Prevents too many requests from exhausting the database connection pool.

• CPU-heavy tasks: Limits expensive computations so the server does not become overloaded.

• Slow downstream services: Protects internal services or third-party APIs from too many simultaneous calls.

• File processing: Controls how many uploads, conversions, or background jobs run at once.

• Backend protection: Keeps active request load within a safe operating range.

It is especially helpful when requests have variable processing time. A small number of slow requests can consume resources for a long time, even if the request rate is not very high.

Advantages and Limitations

Concurrency Limiter gives direct control over active resource usage.

• Prevents overload: The server does not process unlimited simultaneous requests.

• Protects limited resources: CPU, memory, database connections, and external APIs remain safer.

• Works well with slow requests: Long-running requests occupy slots until they finish.

• Supports queueing: Extra requests can wait instead of being immediately rejected.

• Can increase waiting time: Queued requests may experience higher latency.

• Needs careful slot sizing: Too many slots can overload the system, while too few slots can reject valid traffic.

• Does not replace rate limiting: A user may still send too many short requests unless request-rate limits also exist.

In many systems, concurrency limiting and rate limiting are used together. Rate limiting controls request frequency, while concurrency limiting controls how much work runs at the same time.

Summary

Concurrency Limiter is a rate limiting algorithm that restricts the number of requests executing simultaneously. It uses slots to represent available processing capacity. A request occupies a slot while it runs and releases the slot after completion.

If all slots are occupied, new requests are either rejected or placed in a waiting queue. This makes Concurrency Limiter useful for protecting backend servers, CPU, database connections, downstream services, and third-party APIs from overload. It limits active execution rather than request frequency, so it complements traditional rate limiting algorithms.