Giving every user a free daily token allowance

ChatGPT hands every account a daily token budget that differs per model. It is a few hundred lines, and the interesting parts are the concurrency bug, the fact that you cannot know the cost until after you have paid it, and what should happen at zero.

By Jithin Roy

Full Stack Developer, Kerala, India

19 October 20269 min read

This one is a design rather than a write-up. I have not shipped it. Everything here is either a documented API shape or a mechanism I would stand behind, but there are no measured numbers, and I would rather say so than imply otherwise.

ChatGPT now gives every account a free daily token allowance, and it differs by model: millions of tokens on the cheap ones, a couple of hundred thousand on the expensive ones, refreshing on a schedule. It is a genuinely good piece of product design, because it lets someone use the thing generously without a credit card while keeping the operator's exposure bounded.

If you run an AI feature, somebody is going to ask why yours cannot do that. This is what it actually takes.

The number is already in the response you are discarding

The most common reason people do not track tokens is the belief that they would have to estimate them, which means shipping a tokeniser, keeping it in step with the model, and accepting that the answer is approximate.

You do not have to estimate anything. Every provider tells you exactly what you spent, in the response you already received:

const completion = await openai.chat.completions.create({ /* ... */ })

completion.usage
// { prompt_tokens: 1832, completion_tokens: 214, total_tokens: 2046 }

// Gemini returns the same information under a different name:
// response.usageMetadata
// { promptTokenCount, candidatesTokenCount, totalTokenCount }

Most code reads the message content off that object and throws the rest away. Reading two more fields and writing them somewhere is the entire foundation of everything below. It is measurement, not estimation, and it costs one line.

Counting requests protects the wrong thing

Nearly every AI feature that has any limit at all limits requests. Ten messages per session, a hundred per day. It feels like a control and it is barely correlated with what you are trying to control.

Consider two requests. One is a short question with no history and no retrieved context: a few hundred tokens. The other is the twelfth turn of a conversation, carrying a full history window and several thousand tokens of retrieved documents, generating a long answer: perhaps thirty times the cost. Your limiter counts them identically.

Which means the limit is calibrated for the expensive case and is therefore far too strict for the common one, or it is calibrated for the common case and provides no protection at all in the expensive one. There is no setting that is right for both, because requests are not the thing you are being charged for.

Request limits protect your server from load. Token limits protect your invoice. They are different jobs and neither substitutes for the other. Most systems have the first and believe they have the second.

A token is not a token

Once you count tokens you immediately hit the reason ChatGPT's allowances differ per model: a token on a cheap model and a token on a frontier model differ in price by more than an order of magnitude. A single pooled token budget would let someone burn their entire allowance in a few requests on the expensive model, or would have to be set so low that the cheap model becomes useless.

Two workable approaches. Hold a separate bucket per model, which is what the tiered allowance is doing and is the easiest to explain to a user. Or hold one budget denominated in cost and convert on the way in:

// One budget, denominated in cost. Weights are the per-million prices,
// normalised against the cheapest model you offer.
const WEIGHTS = { cheap: 1, mid: 8, frontier: 40 }

const charge =
  usage.prompt_tokens * WEIGHTS[tier].input +
  usage.completion_tokens * WEIGHTS[tier].output

Note that input and output are priced differently, usually by a factor of three or four, so a single per-model weight is already a simplification. Whichever you choose, put the date next to the numbers in a comment. Model prices move, and a weight table with no provenance is a landmine for whoever reads it in a year.

Where the counter lives, and why in-memory is not a counter

The obvious first implementation is a map in module scope. It is also, on any modern deployment, close to useless. I can say that with some confidence because the chat endpoint on this very site does it:

const messageCounts = new Map<string, number>()   // module scope, serverless runtime

const userKey = `${clientIP}-${sessionId}`
const currentCount = messageCounts.get(userKey) || 0
if (currentCount >= 10) {
  return Response.json({ error: 'limit_reached' }, { status: 429 })
}
messageCounts.set(userKey, currentCount + 1)

Three separate problems in eight lines, and it looks completely reasonable.

It resets on every cold start. It is not shared between concurrent instances, so the effective limit is ten times however many instances happen to be warm. It never expires, so in a long-lived process it is also a slow memory leak. And the key includes a session id supplied by the client, which means a visitor gets a fresh allowance by generating a new one.

The fix is to put the counter somewhere durable and shared, and to key it on something the visitor cannot choose. Redis is the natural fit, because the expiry is the reset:

const key = `quota:${userId}:${utcDay()}`
const used = await redis.incrby(key, charge)
await redis.expireat(key, nextResetEpochSeconds())   // set once, on creation

A relational table works too and buys you history you can chart, at the cost of a write on the hot path. Either is fine. A process-local map is not.

Read, check, increment is a race

This deserves its own section because the broken version is the version everybody writes first, and it fails only under exactly the load you built the limiter for.

Read the current usage, compare it against the allowance, and if there is room, write the new total. Two requests arriving together both read the same value, both see room, and both proceed. Under normal traffic you never notice. Under an automated burst, which is the case that costs money, the limit leaks badly.

// Wrong: three steps, no atomicity.
const used = await store.get(key)
if (used + charge > allowance) return refuse()
await store.set(key, used + charge)

// Right: increment first, then decide on the returned value.
const used = await redis.incrby(key, charge)
if (used > allowance) {
  await redis.decrby(key, charge)   // hand it back
  return refuse()
}

Increment and read the result in one operation, then decide. If you are using a database rather than Redis, the equivalent is an update returning the new value, or a transaction with the row locked. The rule generalises: any check-then-act against shared state needs the check and the act to be the same operation.

You cannot know the cost until after you have paid it

This is the genuinely interesting part of the design, and the part no article about rate limiting prepares you for.

You can count prompt tokens before you call, roughly. You cannot know completion tokens until the reply exists, and by then you have already been charged for them. So the decision to allow a request has to be made without knowing what it will cost.

Two ways to handle it, and the right one depends on how you deliver responses.

Charge on settle. Let the request through if there is any budget left, run it, then deduct the actual usage. Simple, and the worst case is that a user with a hundred tokens remaining issues a request that costs two thousand and ends up slightly negative. For a free tier that overshoot is usually acceptable, and it is bounded by max_tokens anyway.

Reserve and settle. Deduct the worst case up front, which is prompt tokens plus max_tokens, then refund the difference once the real usage comes back:

const reservation = estimatedPromptTokens + MAX_COMPLETION_TOKENS
const used = await redis.incrby(key, reservation)
if (used > allowance) {
  await redis.decrby(key, reservation)
  return degrade()
}

try {
  const completion = await callModel()
  const actual = completion.usage.total_tokens
  await redis.decrby(key, reservation - actual)     // refund the unused part
  return completion
} catch (err) {
  await redis.decrby(key, reservation)              // a failed call costs nothing
  throw err
}

The error path is the half people forget. If the model call fails, the reservation must be released, or a provider outage silently eats every user's allowance for the day and the feature stays broken long after the provider recovers.

Streaming forces the choice. With a streamed response the usage figures arrive at the very end, after the user has already read most of the answer, so there is no possibility of deciding partway through. Reserve and settle is the only honest option there.

Make the reset explicit

Store when the window opened rather than inferring it from process state. A counter that resets because something restarted is not a daily allowance, it is a deploy-shaped lottery, and users notice that they get more when you ship.

A fixed hour in UTC is easier to reason about and much easier to explain than a rolling twenty four hours. Rolling windows are fairer in principle and produce a support conversation that starts with well, it depends when you sent your first message yesterday. Pick the one you can put in a sentence: your allowance resets at midnight UTC.

What happens at zero is a product decision

Refusing is the lazy answer, and it wastes the main advantage of thinking in tokens rather than requests. Once you know cost, you can spend less instead of stopping.

Fall back to a cheaper model. The answer is slightly worse and the feature still works.
Shorten the retrieved context. Two chunks instead of five is a large saving and often no worse.
Drop the conversation history window to the last turn or two.
Reduce max_tokens, so answers get more concise rather than absent.

That ladder is why a tiered allowance is a better design than a single limit. The user keeps working, on a smaller engine, and the moment they hit the cheapest tier's ceiling is much further away than the moment they exhaust a single pooled budget.

And whatever you do, say so. A feature that quietly gets worse is a bug report. A feature that says you have used today's allowance, so answers will be shorter until midnight is a product.

Two ceilings, not one

A per-user cap stops one person monopolising the feature. It does nothing about a thousand people, or one person with a thousand accounts, and that is the scenario that produces a four figure invoice overnight.

A global daily ceiling is a handful of lines and is the only thing standing between an automated attack and your card. Set it well above legitimate usage, alert on approach, degrade on breach. The per-user cap is about fairness; the global cap is about survival. Neither is sufficient alone.

What to take from this

Read usage off the response. Both major providers return exact token counts you are probably discarding.
Count tokens, not requests. Requests protect your server; tokens protect your invoice.
Weight by price or bucket by model. A token on a frontier model is not a token on a cheap one.
Put the counter somewhere durable and shared, and never key it on something the client controls.
Increment atomically and decide on the returned value. Read-check-write leaks under exactly the load you built it for.
Reserve the worst case and refund the difference, especially when streaming, and always release the reservation on failure.
Store when the window opened, and prefer a fixed UTC hour you can explain in one sentence.
Degrade instead of refusing: cheaper model, less context, shorter history, smaller max_tokens. Then say so.
Add a global daily ceiling alongside the per-user one.

The reason to build any of this is not really cost control. It is that knowing what things cost lets you be generous on purpose instead of cautiously stingy by default. A limit you understand can be set high. A limit you do not understand gets set low and quietly makes the feature worse for everybody.

If I implement this on the widget here, I will publish the measured version: real cost per conversation, and how it splits between embedding, retrieved context and generation. Until then, treat the above as a design and not a report.