We had a problem over at Design Week recently. The server was groaning a little bit and a huge amount of bandwidth being eaten up. On investigation, it seemed to be a bot that was stuck in a loop, reading the same content again and again and again. Always the same handful of URLs.
This isn’t an entirely unusual occurrence on our publisher sites. Generally we have enough capacity that we don’t notice, but there’s one really uncomfortable truth. For every webpage we serve to a human, we serve 4 or 5 to a bot. Some of those are indexing the site and looking for changes, like Google or Bing, and others are bots slurping up everything on the site, requesting over the course of a day all 58000+ articles on the website. Which often means they’re requesting from the origin server.
And this feels a little… unfair. And damaging. Bigger servers cost us more money. We spend hundreds each month on serving Design Week, plus all the monitoring, backups, etc. It’s a substantial site with substantial traffic.
Oddly, viral traffic is far less of a problem – if a million people want to see a particular page our origin servers barely wake up, because the cdn just serves caches of pages to nearby visitors. But if four or five greedy bots turn up they soon overwhelm caching limits and use up a lot of resource.
For what? To feed their business model? So they can offer up rehashed versions of our content to their users so they don’t have to visit our website any more? Well what’s the point of that for us. The end user isn’t paying us. The AIs aren’t paying us. The advertisers won’t pay for robotic views.
Meanwhile, it costs us between £50 and £300 to produce a well written and good article on the site, depending on the nature of it. Short opinion pieces are easy, but we don’t do many of those. Some are contributed for free as a way of pushing a campaign idea or agenda that the author believes in and which chimes well with readers.
AI is eating its own children. It relies on authentic, human written and unique content. And search engines are also no longer sending people to websites, but providing a summary of the content on the sites they’ve indexed. It reaches a point where it’s almost tempting to cut off every bot and put all content behind a paywall and just step away from the freeloaders.
I’m a big believer in the open web. That knowledge should be free to circulate. But I also have a mortgage to pay and kids to feed. I don’t get to moralise about the importance of open information when I’ve gained some nice big share options funded by venture capitalists desperate for a huge return based on the work of millions of others.
At the same time, I really do think AI in its current form is incredibly useful. I’ve used it to ideate and prototype a new brand direction for our company, to flesh out ideas so that I can show something closer to what’s in my head before handing the idea over to People Who Know How. I’ve vibe coded WordPress plugins, illustrated a brochure, and checked contracts for inconsistencies or problematic clauses quickly and easily. And when that brochure was seen as a good idea, it felt worth commissioning an artist to do real illustrations because ultimately people are much better at creative work.
But I’m uncomfortable with more tech billionaires being created by leaching off the back of people’s work when many writers, journalists and publications are struggling to put a living together.
Inevitably, something has to break. We block them. Or we charge them. I don’t see how anything else could work. What do you think? Comments are open.
Illustration by DALL·E because I’m not short of irony and hypocrisy.
