Security
Rate limiting will not stop a scraper
A competitor who can enumerate your prices owns your pricing strategy. Rate limits do not stop them. What does is making the attempt cost more than the data is worth, and that is an economics problem before it is an engineering one.
Every guide to stopping scrapers opens the same way: install a rate limiter, set a requests-per-minute ceiling, done. It is the first thing you should do and it is nowhere near sufficient, because a rate limit does not stop a scraper. It sets the speed at which the scraper works.
Somebody willing to wait will get your data. The question is never how do I make this impossible. It is how do I make it cost more than the data is worth. That reframing changes what you build, and it is why the most effective control in the system I want to describe is not a technical one at all.
Work out what is actually valuable
The site in question buys used phones. It has a catalogue of devices, which is public and uninteresting. Anybody can learn that the business buys iPhones. What is valuable is the quote: the exact figure the business will pay for a specific variant in a specific condition.
A competitor who could enumerate quotes across the catalogue would not have scraped a price list. They would have reconstructed the pricing strategy for every device the business buys, including the depreciation curves and the condition weightings, all derivable by varying one input at a time. That is the crown jewels, and it is not a document anywhere. It is the output of a form.
So quoting is treated as the sensitive operation and everything else is treated as public. That decision, made first, is what makes the rest of the design coherent. Protect the endpoint that produces value, not the pages that describe it.
Why one threshold cannot work
Consider what a genuine customer does. They are clearing out a drawer. They quote their old iPhone, then their partner's Android, then a tablet nobody has used in two years. They get a number they dislike, go back, and change their answer about the screen to see how much that mattered. That is a dozen quotes in ten minutes from one person acting entirely in good faith.
Now set a limit low enough to stop a scraper. You have just stopped that customer. Set it high enough for that customer and a script running quietly under it will walk the whole catalogue in a weekend. There is no single number that separates the two, because at small scale the behaviours genuinely overlap.
The way out is to stop asking one question. Use two windows with different consequences:
const RAPID_QUOTES_THRESHOLD = 10 // in
const RAPID_WINDOW_MINUTES = 5 // minutes -> blocked outright
const SUSPICIOUS_QUOTES_THRESHOLD = 30 // in
const SUSPICIOUS_WINDOW_MINUTES = 60 // minutes -> allowed, but reportedTen quotes in five minutes is not shopping, it is a loop, and it gets blocked. Thirty in an hour is unusual but not impossible for somebody clearing out a family's worth of old devices, so it is allowed and a human is told about it. The customer above trips neither. Sustained abnormal volume reaches a person rather than a wall.
The useful question is not what limit stops abuse. It is which behaviours should be blocked by a machine and which should be judged by a person. Those are different thresholds and they deserve different consequences.
The response depends on who you are, and getting this wrong bans customers
This is the part that is easy to implement carelessly and expensive to get wrong.
Anonymous traffic that trips the blocking threshold gets its IP banned. A signed in account that trips it gets suspended instead, and its IP is left alone. That asymmetry is deliberate, and the reason is mobile networks.
In India, most traffic is mobile, and carrier grade NAT means a single public IP can front thousands of subscribers. Ban that address because one person ran a script and you have quietly blocked everyone else behind it, in a city, on that carrier, with no error message that tells them why. You will not find out. They will just stop being customers.
When somebody is signed in you have a far better identifier than their network location, so use it. Suspend the account, leave the address alone, and the collateral damage goes to zero.
if (identifierType === 'ip') {
await db.query(
"INSERT INTO bans (ip_address, reason, ban_type) VALUES (?, ?, 'ip')",
[identifier, 'Rapid quote creation - automatic temporary ban']
)
} else if (identifierType === 'user_id') {
await db.query('UPDATE users SET suspended_at = NOW() WHERE id = ?', [identifier])
}Normalise before you match, or the ban does nothing
A ban is a string comparison, and strings for the same human being are not equal. An Indian mobile number can arrive as ten digits, with a plus nine one, with a bare nine one, with spaces, with a hyphen. Ban one spelling and the same person walks back in through another.
function normalizePhone(phone) {
if (!phone || typeof phone !== 'string') return null
const digits = phone.replace(/\D/g, '')
return digits.length >= 10 ? digits.slice(-10) : (digits || null)
}
// ...and the lookup still checks the stored variants, because rows
// written before normalisation existed are still in the table.
'SELECT id, reason FROM bans WHERE (phone = ? OR phone = ? OR phone = ?) AND ...'
// [normalized, `+91${normalized}`, `91${normalized}`]Emails get the same treatment, lowercased and trimmed. The general rule: any identifier you ban on needs one canonical form, decided once, applied on write and on read. Bans are one of the few places where being strict about input format is not pedantry, because a near miss is a silent failure.
Bans are records, not config
Every trip writes a report before anything is enforced. The count, the window, the identifier and its type, and a status:
CREATE TABLE quote_abuse_reports (
identifier_type enum('user_id','session_id','ip') NOT NULL,
identifier_value varchar(255) NOT NULL,
abuse_type enum('rapid','suspicious') NOT NULL,
quote_count int NOT NULL DEFAULT 0,
window_minutes int NOT NULL DEFAULT 0,
status enum('pending','reviewed','banned','cleared') NOT NULL DEFAULT 'pending',
admin_notes text DEFAULT NULL,
resolved_at timestamp NULL DEFAULT NULL,
resolved_by_admin_id int DEFAULT NULL
)Two things follow from this that would not follow from a rate limiter holding counters in memory.
First, you can answer the question. Somebody will eventually email to say they cannot use the site, and they will be a real customer whose behaviour looked wrong. If you cannot pull up the count, the window and the timestamp that caused it, you cannot tell whether the system was right. You will guess, and half the time you will guess wrong.
Second, the review workflow is a workflow. Reports move through pending to reviewed, banned or cleared, with notes and the admin who resolved them. A suspicious report for an identifier that already has one pending updates that row rather than inserting another, so a single busy afternoon produces one item in the queue instead of forty. Nobody reviews an alert queue that cries wolf.
The control that actually works is economic
Everything above is friction. Here is the part that changes the arithmetic.
Getting a quote at any scale requires an account. An account requires a verified mobile number. And the blocking threshold trips long before one number has paid for itself.
Work it out from the attacker's side. A catalogue of a few thousand variants, each needing several quotes to map its condition curve, against a ceiling of ten quotes per five minutes per identity, with the account suspended the moment it trips. That is not a scraping problem any more, it is a procurement problem: it takes hundreds of working phone numbers, each of which costs real money and real effort to obtain, to extract a price list that is worth considerably less than that.
No amount of cleverness in the client makes that cheaper. The attacker cannot rotate their way out of it, because the scarce resource is not addresses, it is verified identities. That is the whole defence, and the code above exists mostly to make sure it cannot be bypassed.
Obfuscation buys hours, not safety
Public API responses go through a key map on the way out. Every field name is replaced with a short code, so a response reads as rv, i, n, rt rather than reviews, id, name, rating.
const skipObfuscate =
req.path.startsWith('/admin') ||
req.path.startsWith('/agent') ||
req.path.startsWith('/api/auth') ||
req.path.startsWith('/api/upload') ||
req.path === '/health' ||
req.headers['x-client'] === 'mobile'
res.json = function (body) {
return orig(skipObfuscate ? body : obfuscate(body))
}Be clear about what this is. It is not encryption and it is not security. Anyone who spends twenty minutes with the network tab has the mapping. What it does is raise the cost of the cheapest attack, the person pointing a generic scraper at an endpoint and grepping for a field called price, and it makes casual reuse of the API by third parties annoying enough that they ask instead.
The exclusions matter as much as the rule. Admin, agent, auth and upload paths are exempt, so internal tooling stays readable and debuggable, and so is anything sending the mobile client header. A security measure that makes your own systems hard to operate gets switched off during the first incident and never switched back on.
Where this is deliberately weak
Three things, stated plainly, because a security write-up that claims no gaps has not been read carefully.
The ban checks fail open. Every one is wrapped so a database error returns not banned rather than throwing. That is availability chosen over enforcement: a database blip should not lock every customer out of the site. It also means a sufficiently unlucky moment lets a banned identity through. For this threat model that is the right trade, and it is a trade, not an oversight.
Counting happens against the quotes table rather than an in-memory counter, which costs two queries on the hot path. In exchange the counts survive restarts, are shared across every instance, and are the same numbers a human sees when reviewing the report. An in-memory limiter in a horizontally scaled deployment is mostly decorative, and the cost of getting that wrong is that you believe you are protected when you are not.
And none of this stops a well funded adversary who genuinely wants the data. Someone prepared to buy hundreds of verified numbers and drip requests below every threshold will get it. That was never the goal. The goal is to move the cost above the value for everybody else, which covers essentially every real attempt.
What to take from this
- Identify the operation that produces value, not the pages that display it, and protect that.
- Use two thresholds. One a machine enforces, one a person reviews. A single limit has to be wrong for somebody.
- Never ban a shared IP when you have an account to act on. Carrier NAT means one address can be thousands of customers.
- Normalise every identifier you ban on, once, on write and on read. A near miss is a silent failure.
- Write evidence before you enforce, and give reports a review workflow. You will have to justify a ban to a real person.
- Deduplicate alerts. A queue that fires forty times for one incident stops being read.
- Put the real defence in the economics: make the scarce resource something that costs the attacker money.
- Treat obfuscation as friction, exempt your own tooling from it, and never mistake it for a control.
- Decide consciously whether your checks fail open or closed, and be able to say which and why.
The uncomfortable part of writing this down is that the most effective control took the least code. Requiring a verified phone number for the valuable operation did more than every threshold, ban table and key map combined. The engineering exists to make sure that requirement cannot be walked around. It is worth asking, before building any of it, what your equivalent of that is. The admin side of this, including the review queue and the permission model behind it, is covered in the SellPrime case study.