Token Bucket Rate Limiting Algorithm

Introduction

Token Bucket is a flexible rate limiting algorithm used to control how quickly requests are allowed to enter a system. It protects backend servers and APIs from continuous overload while still allowing short bursts of traffic when the system can handle them.

The key idea is simple: a request can pass only if a token is available. Tokens are added to a bucket at a fixed rate, and each request consumes one token.

What Is Token Bucket?

Token Bucket is based on a logical bucket that stores tokens. The bucket has a maximum capacity, which decides how many tokens it can hold at once.

For example, a bucket may hold:

• 100 tokens

• 500 tokens

• 1,000 tokens

• 10,000 tokens

The actual bucket size depends on the system requirement. A larger bucket allows bigger short bursts, while a smaller bucket keeps traffic more controlled.

How Token Bucket Works

The algorithm has two important settings:

• Bucket capacity: Maximum number of tokens the bucket can store.

• Refill rate: Speed at which new tokens are added to the bucket.

A simple request flow looks like this: Request arrives => Token is checked => Token is consumed => Request is allowed

If no token is available, the request may be rejected or made to wait, depending on the implementation. In API rate limiting, rejected requests are commonly returned with: HTTP 429 Too Many Requests

The basic working is:

• Bucket is created: A fixed-capacity token bucket is maintained.

• Tokens are refilled: New tokens are added at a fixed rate.

• Request arrives: The rate limiter checks whether a token exists.

• Token exists: One token is consumed and the request is allowed.

• No token exists: The request is rejected or delayed.

Why Token Bucket Allows Bursts

Token Bucket allows bursts because unused tokens can accumulate during idle time. If the bucket becomes full, many requests can be allowed immediately.

For example, if no requests arrive for some time, the bucket may collect 1,000 tokens. If 1,000 requests suddenly arrive, they can be allowed because tokens are already available.

However, this burst cannot continue forever. Once the bucket becomes empty, new requests must wait for fresh tokens or get rejected.

So Token Bucket provides two controls at the same time:

• Burst capacity: Controlled by the maximum bucket size.

• Long-term average rate: Controlled by the refill rate.

Token Bucket vs Leaky Bucket

Token Bucket and Leaky Bucket are both used for traffic control, but they behave differently with bursty traffic.

Token Bucket is more flexible because it allows temporary spikes. Leaky Bucket is stricter because it focuses on smoothing traffic into a steady output.

Advantages and Limitations

Token Bucket is popular because it matches real traffic behavior better than algorithms that reject every sudden spike immediately.

• Allows controlled bursts: Useful when short request spikes are acceptable.

• Enforces average rate: Refill rate prevents continuous abuse.

• Flexible configuration: Bucket size and refill rate can be adjusted separately.

• Useful for APIs: Works well for API throttling and backend protection.

• Can still allow sudden load: A full bucket may permit a large burst at once.

• Needs careful tuning: Wrong bucket size or refill rate may be too strict or too loose.

The algorithm is strong when occasional bursts are valid, but it should be configured carefully so bursts do not overwhelm downstream services.

Summary

Token Bucket is a rate limiting algorithm where tokens are added to a bucket at a fixed refill rate, and each request consumes one token. If a token is available, the request is allowed. If no token is available, the request may be rejected or delayed.

Its main strength is flexibility. It allows short bursts when tokens have accumulated, but still enforces an average request rate over time through the refill rate. This makes Token Bucket useful for API rate limiting, traffic shaping, request throttling, and backend abuse prevention.