Ironic ChatGPT use case

342 words, 2 minutes.

I used to quite enjoy writing regexs. man perlre is one’s friend. However, sometimes you have to realise there’s a quicker way.

This week, a good friend and I have been having a lengthy chat, over texts, spanning a few days. We’ve been discussing the unfortunate ‘dumbing down’ of the tech industry, and how ’environment determines behaviour’¹ — the more abstracted and layered in frameworks things become, the less engineers actually understand what’s happening underneath it all. The less they understand, the less they seemed to want to understand. And so skills are vanishing.

So with a massive dollop of irony, I asked ChatGPT to do something for me that ordinarily I’d a) do with relative aplomb b) probably enjoy.

For a while now I’ve had a blocklist on my web server configuration that blocks known AI bot scrapers. I convert the robots.txt list into a configuration line for nginx, as per Robb Knight’s blog post.

Every time the list gets updated, I change the string in nginx. But today I couldn’t be bothered to work out what had changed, and just wanted to copy & paste the list. I was about to put the list into Vim and record a quick macro, when it dawned on me I should probably write a script and submit it to the repo. But then I thought, “I wonder if ChatGPT can sort this quicker?”

Answer, yes it can:

Hey! Can you take this list and remove "User-agent:\s" then join every entry together with | please?
...
Here’s the list with “User-agent: “ removed and all entries joined together with |:

Finding this irony a touch comical, I posted it to the discussion area of the GitHub repo. That conversation prompted a response, and a question that caused me to further dumb myself down:

https://github.com/orgs/ai-robots-txt/discussions/34

So there we go. A one-liner awk is the result:

awk '{gsub(/^User-agent: /, ""); s = s $0 "|"} END {sub(/\|$/, "", s); print s}' user_agents.txt