Fast(er) regular expression engines in Ruby

SerpApi | https://serpapi.com | Junior to Senior Fullstack Engineer positions | Customer Success Engineer | and more... | Based in Austin, TX but remote-first structure | Full-time | FULLY REMOTE or ONSITE | $150K - 180K a year 1099 for US or local avg + 20% for outside the US

SerpApi is the leading API to scrape and parse search engine results. We deeply support Google, Google Maps, Google Images, Bing, Baidu, and a lot more.

We are still hiring - we have so much work and we're after great people to help out! I only started at SerpApi late last year myself and can't recommend it enough.

We do continuous integration, continuous deployments, code reviews, code pairings, profit sharing, and most of communication is async via GitHub.

Our current stack is Ruby, Rails, MongoDB, and React.JS. We are looking for more Junior and Senior FullStack Engineers. We have an awesome work environment: We are a remote first company (before Covid!).

We very strongly value transparency, do open books, have a public roadmap, and contribute to the EFF.

When you apply, please mention that you saw David's post on HackerNews.

Apply at: https://serpapi.com/careers

If you've previously applied and didn't make it through, please feel free to reapply if it's been a while and you think you would make a better fit than previously.

You can contact me at david at serpapi.com if it's been a few days and you think your application may have fallen through the cracks.

Making it harder for bots usually means that it drives up the cost for the bots to operate; so if they need to run in a headless browser to get around the anti-bot measures it might mean that it takes, for example, 1.5 seconds to execute a request as compared to the 0.1 seconds it would without them in place.

On top of that 1.5 seconds is also that there is a much larger CPU and memory cost from having to run that browser compared to a simple direct HTTP request which is near negligible.

So while you'll never truly defeat a sufficiently motivated actor, you may be able to drive their costs up high enough that it makes it difficult to enter the space or difficult to turn a profit if they're so inclined.

Very impressive work! I always enjoy a good write up about reverse engineering efforts and yours was really simple to follow.

Many popular/large websites and bot protection services usually have environment checking as a baseline and mouse-movement tracking in some of the more aggressive anti-bot checks.

It's always interesting to see how long it takes from when the measures have been defeated/publicised until the service ends up making changes to their mechanism to make you start over (hopefully not from scratch).

Some link shortening services allow you to either bring your own domain or register a domain through them to use, so in the latter case it would be "via" the link shortener.

It's sometimes available on other types of SaaS services where your URL would otherwise be customer-facing as well.