Using HAProxy to block scrapers

  • 5 min read

I used to protect some services hosted on chatoyer.de with Proof-of-Work challenges, namely using Anubis and go-away. Such challenges can be solved quite quickly on modern desktop hardware. However, less powerful devices, like my 200€ smartphone, take multiple seconds to solve the challenge. On the other hand, professional scrapers are often located in large datacenters with much more compute power than any normal user has. In the end, this kind of challenge achieves exactly the opposite of what I want to achieve: Instead of blocking bots and making the service available to normal users, scrapers can pass almost immediately and normal users have a higher hurdle for using the website (usually multiple seconds of time waiting). Tavis made the math in their blog post: Scrapers can solve the challenge within few milliseconds [1].

There are two types of scrapers: the ones that run a full browser, and the ones that only make simple HTTP requests (e.g. with curl). It is not realistic to stop the first type from accessing the website with challenges like Anubis. They can execute JavaScript just as normal users can do, so the browser automatically solves the challenge for them. Anubis primarily blocks the second type, i.e. scrapers that don't run in a browser and can't execute JavaScript. But in this case, Proof-of-Work challenges are quite of an overkill, it's sufficient to just use any other technique to confirm that the website is accessed via a real browser.

Idea

There's no good solution for blocking scrapers that run a full browser, so I decided that the best solution that doesn't waste the user's time (like Anubis does) is to only block the non-browser scrapers. The implementation is very much inspired by the awesome blogs posts from David Leatbeater and flexagoon. Essentially, we ask the client to verify that it's a browser by letting it proof that it supports persisting cookies. In short, we set a cookie on the client's device and only let the it access the page if it includes that cookie on the next request.

graph LR
    A[HTTP request] --> B[challenge page]
    B -. sets cookie and refreshes .-> C[actual website]
    B -. cookie not stored .-> B

Yes, non-browser scrapers can also remember cookies and send them on the next request, but they usually just send a single HTTP request without any content and don't handle cookies. The advantage of this over Anubis' or go-away's meta-refresh challenge is that it can be integrated directly into HAProxy, so we don't need to host a third party service in front of the web service we want to protect.

Implementation

We need two parts:

  • the HAProxy configuration haproxy.cfg that generates and validates the cookie
  • the challenge page refresh-challenge.html, which is shown while the user waits for the page to refresh
frontend http
  bind :80 v4v6

  # build client fingerprint and store it in txn.client_hash
  http-request set-var(txn.ip) src
  http-request set-var(txn.secret_key) str(supersecretkey)
  http-request set-var(txn.client_hash) req.fhdr(User-Agent),concat(;,txn.ip),concat(;,txn.cur_month),digest(sha256),base64

  # show challenge page if not verified (cookie missing or wrong)
  acl is_verified req.cook(verified),strcmp(txn.client_hash) eq 0
  use_backend meta-refresh if !is_verified

  default_backend my_backend

backend meta-refresh
  http-after-response set-header Set-Cookie "verified=%[var(txn.client_hash)]; path=/"
  http-request return content-type text/html file /etc/haproxy/refresh-challenge.html

backend my_service
  http-request return status 200 content-type "text/plain" string "Passed challenge"

The prefix in front of the variable name determines the scope in which the variable is available. For example, txn means that the variable is available during the entire HTTP request-response transaction. [2]

We configure HAProxy to calculate a client fingerprint. In this case it consists of the request's User-Agent, IP and a server-side secret key. IP and user agent are used to make the hash result unique per client. The server-side secret key prevents clients from generating the cookie themselves without ever viewing the challenge page. Technically that's not needed, because scrapers probably don't know how I compute the hash (except if they read this blog post :o). The fingerprint could also be extended to contain a date (e.g. so that the cookie is only valid one day) or other fields, but I decided to keep it simple. Note that everything is hashed using SHA256 in the end, which means that it is not possible to reverse the process, i.e. derive the IP and user agent from the final client fingerprint.

Upon a request, HAProxy simply checks if the verified cookie is set and matches the client fingerprint. If yes, we return the site the user requested. If not, we return a response that sets the verified cookie to the expected value and show the challenge page, refresh-challenge.html.

<!DOCTYPE html>
<html>
  <head>
    <!-- ask browser to immediately reload the page -->
    <meta http-equiv="refresh" content="0">

    <!-- boilerplate to fix scaling on mobile devices -->
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <!-- let browser automatically show a dark version of the page in dark mode -->
    <meta name="color-scheme" content="light dark">

    <title>Verifying ...</title>
  </head>

  <body>
    <h3>Verifying ...</h3>
  </body>
</html>

The magic part in refresh-challenge.html is <meta http-equiv="refresh" content="0">. This causes a normal browser to refresh the page immediately when it finished loading [3]. Because the response that returned refresh-challenge.html also returned a Set-Cookie header to ask the browser to store the verified cookie, the browser sends the verified cookie when refreshing the page. HAProxy sees that the cookie is correct and shows the response from the actual service the user requested (my_service in this example).

Closing Words

If you're interested, you can see the challenge in action here. So far, it performed quite well for me, i.e. I didn't run into any ratelimits on any of the services I protect with this challenge. Of course, this is nowhere as reliable as Cloudflare CAPTCHAs, and you should still employ IP-based ratelimiting to prevent malicious actors from running DoS attacks on services. The main selling point for me is that this solution is significantly faster than Anubis and wastes less of the user's (and my own) time waiting for the challenge to finish. The time it takes to get past the challenge is only limited to the network speed (i.e. for reloading the page), but not on the client's hardware performance; it works completely without JavaScript.

  1. https://lock.cmpxchg8b.com/anubis.html ↩

  2. https://www.haproxy.com/documentation/haproxy-configuration-tutorials/proxying-essentials/custom-rules/variables/ ↩

  3. https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/http-equiv ↩