“Don’t make me think”.

Steve Krug¹

There’s something I say all the time when advising startups on their product strategy: solve problems for people, and the money will come². It still amazes me that this often comes as a shock.

All too often companies try to ‘game’ their customers: bait ’n switches, enshittification, bullshit marketing. These are all signs of a company focussed more on itself than its customers. They somehow convince themselves that being customer focussed is a lesser path to profits ³.

I can hear you snorting now though - “you’re being idealistic and unrealistic!” Yet throughout my 30+ year career, I’ve seen behaviours that prove the point: make a product that solves a customer problem, make it easy for them to love you for solving the problem, and the revenue comes. Even if you’re not charging for the product.

A couple of weeks ago, I saw another proof point of this, with a product I have only recently discovered, yet already love enough to want to pay for it, although I don’t have to pay for it. Tailscale.

What makes a great product

Here’s how to make a great product that will sell: solve a problem and make it incredibly easy for other people to adopt and use your product to solve their problem. That’s it (again, see ²).

The automatic thought is Tailscale is a VPN, but really, it’s not. It’s a simple way to connect devices securely and privately. Trying to get yourself off the cloud drug and repatriate your own data to a NAS you own, at home, yet want to access it from your phone when away? You need a way to securely and privately connect those devices without exposing your private NAS to the internet. Tailscale is the answer.

However, implementing technical products like this is often hard. Which is where Tailscale hits my simple requirement. I connected a handful of devices with barely any thought whatsoever. There was no need to mess about with firewalls; I didn’t need to think about or understand routing and IPv4 versus IPv6. I just followed their simple instructions and it all just worked.

As I’ve used Tailscale more, I’ve realised they also tick a few more of my required checkboxes. Whilst looking at the interface the other day, I noticed something about “an exit node” — with a small (i) icon for more information. The information box showed me how to make a device an exit node; what switch to run the command with. No need to branch off to the documentation to find out how to do this; the minimal information required was there, in a helpful little box.

Don’t make me think

To make a product really easy to use, the complicated has to be simplified somewhere. A lot of companies don’t put enough effort into that simplification, leaving their users to struggle. This is a perennial problem in the tech industry. However, if you, the company developing a product, put the hard work into that simplification process, the user benefits. Which ultimately means you benefit, because they’re more willing to use a product that has little friction to adopting it.

If I can’t be productive with a product within 15 minutes, I will give up on it. If a piece of information isn’t clear, or even worse, there is deliberate obfuscation that makes me work to understand what I’m looking at, I won’t trust the company. I don’t think I’m alone in this thinking, either. People are tired of being used, trapped and ’experimented on’ just to use a product. The hard work of simplifying has to happen somewhere — and it should be you, not your users, doing it. Make it so easy to use your product that I don’t have to think. Thinking is literally costly to our bodies (it takes more energy to use “System 2” than it does “System 1”⁴).

It starts with company culture

“Company culture” is a woolly phrase, too often used as a euphemism to create tribal loyalty. That’s not what I’m talking about here. I’m talking about leadership’s appreciation and respect for people, not least their customers. When the leadership of a company is driven by the desire to solve problems for people, it more often than not flows through the company to create respectful products.

Which brings me back to Tailscale. Recently they changed their pricing and made their free tier even better. In the live chat on Discord (you can watch the recording here) Tailscale’s CEO, Avery Pennarun⁵, actually said at one point (10 minutes in to the recording) that 99% of customers on the paid-for Plus product don’t use any of its features, they just want to give the company money.

I’ve seen that behaviour before. In the early days of Ansible, we had a customer pay for the commercial product, which they didn’t actually need. They loved the open source part so much they just wanted to give the company some money.

Make life easier, faster, or cheaper

We humans dislike change. Even if we have a problem, there can be resistance to adopting a new product to solve it. Especially since historically products haven’t always solved the problems they purport to. A product needs to solve a problem that, in doing so, makes the user’s life easier, faster, or cheaper. Even better if it nails all three. A person’s inertia is far easier to overcome if trying your product is as simple as possible. Don’t make me think.

Any third-rate engineer can increase complexity; but it takes a certain flair of real insight to make things simple again.

E F Schumacher

Thanks for reading this post via RSS! 👏

The TL;DR is “be very worried, but don’t stick your head in the sand”.

Dario Amodei…has publicly predicted that AI will eliminate 50% of entry-level white-collar jobs within one to five years

The often discussed point about AI replacing jobs is those early-career positions. The juniors fresh out of university, gathering vital experience. If AI replaces these entry-level jobs, where does that leave humans?

Peers further along their careers often mention their use of LLMs for tasks previously handled by junior staff. Possible confirmation of Amodei’s statement.

However, a conversation with a close friend recently presented a contrarian perspective. He thinks early-career-stage staff working with LLMs will be far more useful to many businesses.

As our conversation progressed, I posited I’d be concerned about that idea, because it’s all too easy to take what an LLM says as being ‘the one truth’ (it often sounds convincing, even when it’s not; “hallucinations”) and simply running with that, learning nothing. That is the painful but necessary journey of an early career — we learn far more from our mistakes than our successes. This begged the obvious question, how does one set guidance for juniors anyway?

Considering this, we spoke about possibly using a ‘guiding prompt’ for junior staff interacting with an LLM.

Unsure what the prompt ought to resemble, I asked Claude:

I had an interesting chat with a good friend earlier today. We’re both older, 30+ years into our respective careers. We were discussing how AI, like yourself, is changing careers for young folks. There are thoughts that junior roles are extinct, because the more experienced workers can ask ‘better quality questions’ of LLMs like you — and probably get more useful responses than asking a junior. On the other hand, there is the thought that give juniors a good LLM, and they can instantly become far more useful than their experience would ordinarily suggest.

What I worry though is how do junior staffers get to learn how to ask better quality questions if they’re instantly given answers that sound very plausible, even if they’re not (the infamous ‘hallucination’ problem).

Humans, as you well know, are prone to confirmation bias, it takes a different mind to ask “how can I be wrong?” when presented with information. Tangentially, there’s an interesting metaphor I’m fond of telling people about — from Ed Catmull’s book, Creativity Inc. He talks about art classes for the none-artistic staffers, and how the teacher taught them to draw a chair, don’t try to draw a chair, draw the space around the chair. I find that akin to asking “how can I be wrong?” versus looking for confirmation.

So, given all that waffle, what I’d like you to think about is if you were to write a prompt for juniors to give to yourself, that would help you to steer their thinking, what would that prompt look like? How could you help them, to help themselves, to use you wisely, like somebody with many more years might.

Have a think on it and come back to me later. It’ll be worth looking at second, even third or more, order effects and how that might shape the initial guiding prompt.

I used Opus 4.6 with Extended Thinking on and left Claude to ponder things whilst I made a cup of tea.

The answer is a fascinating read, as is Claude’s idea for a prompt for itself. Here’s a link to the actual chat with it.

I’d be interested in your experience and thoughts on this. Either leave a comment on the LinkedIn post, or feel free to email me via the button below.

In case you didn’t read the whole reply from Claude — and I’d encourage you, because its ’thinking’ is interesting in itself — here’s the prompt it came up with.

You are my thinking partner, not my answer machine. I'm early in my career and I'm learning how to think well, not just how to get solutions quickly. Please work with me in this way:

When I ask you something, start by asking me what I think the answer might be, what I've already considered, and what I'm unsure about. Don't skip this step even if I push back — it's the most valuable part of our exchange. If I can't articulate my assumptions, help me figure out what they are.

When you do respond, be explicit about where you're less confident and where you might be wrong. Tell me specifically what I should verify independently and how — not just "check the docs" but which docs, what to look for, what would confirm or refute what you've told me.

Show me the negative space around my question. What's adjacent to what I asked that I haven't thought about? What constraints, failure modes, or second-order consequences am I likely missing? What would someone more experienced than me probably ask that I didn't?

If my question reveals a misconception, don't just correct it — explain the shape of the misconception so I can recognise similar ones in future. Patterns matter more than individual facts.

When I accept an answer too quickly — especially on something complex — push back. Ask me about edge cases, ask me what could go wrong, ask me how I'd test whether what you've told me is actually correct. Don't let me be lazy about this.

If something is genuinely beyond what you can reliably help with, say so clearly and tell me who or what I should consult instead. A textbook, a senior colleague, a specific community — be concrete.

At the end of substantive exchanges, reflect back not on the answer but on the process. What questions were most productive? What assumptions were worth challenging? What should I go and explore on my own? Over time, I want to need this prompt less because I've internalised these habits.

Thanks for reading this post via RSS! 👏

But with iOS 18 and macOS Sequoia, I’ve become quite disillusioned with Apple’s approach to OS ‘improvements’. There was nothing better over iOS 17 nor over Sonoma. Adding nonsense like ‘Image Playground’ and the utter dog’s dinner that is the Photos app are not my idea of improvements. Siri seems to get worse over time, and Apple Intelligence got turned off almost immediately.

Now they’ve really jumped the shark¹.

The Liquid Glass UI is a mess, especially on Tahoe. With some positively amateur UX mistakes like not being able to grab the corners of a window to resize it 🤦‍♂️

Unfortunately, my wife’s phone automatically upgraded to iOS 26. Here’s one example of terrible user experience, in the camera app.

On iOS 18 it’s clear to see that one ‘rotates’ the mode selector — the other modes fade off to the sides, as if one is looking at a rounded control, side on.

On iOS 26 the signifier suggests there are only two modes, yet in actual fact the ‘rotate’ behaviour is the same. Your automatic reaction is likely to be to press each option, and wonder where the heck the portrait mode and other options went.

What were they thinking? How did this stuff get out of the door?

They can put the annoying little red dot on system settings all they like, but I am not upgrading. And I wish they’d stop pushing Apple News. It’s junk, I don’t want it.

In fact, this is pushing me to find alternatives to everything iCloud, to prepare for the day they make everything bad enough for me to ditch 20 years of Apple use for alternatives.

The thing is, all of this just makes me sad, not angry. Apple used to be more or less the only player in an increasingly rubbish tech industry who used design as a discipline to produce better software. In a world where other software companies thought it sensible to scroll content by pulling the bar down, versus scrolling the actual content, Apple has generally got UX right (let’s ignore the terrible magic mouse design problem)

When long-term Apple fan journalists are refusing to upgrade too, Apple should be seriously taking notice.

I joked last week that it would make more sense if we found out that the team behind redesigning the UI for MacOS 26 Tahoe was hired by Meta not a month ago, but an entire year ago, and secretly sabotaged their work to make the Mac look clownish and amateur. More and more I’m wondering if the joke’s on us and it actually happened that way. It’s like MacOS, once the crown jewel of computer human interface design, has been vandalized.

Biting.

UPDATE 11-Feb-26: It looks like some clever folks have identified ways to stop macOS nagging to upgrade. I’ve just run this, and it works a treat.

Thanks for reading this post via RSS! 👏

Although I have stuck to the practice since 2017, I have been through phases where I’ve not shown up for weeks. Last year though, I was determined to stick to more regular sessions. This turned out to be three to four times each week.

I don’t use an app any more, there’s no need, but I do log sessions in the Health app on my phone. I use a simple countdown timer, usually for 20 minutes.

Last year then, I logged 3,304 minutes, a little over 55 hours.

And it was enough to experience profound effects on far more frequent occasions than ever before.

I’ve concluded that meditating is a lot like physical exercise — use it or lose it, basically. But once one has built a solid ‘base’, it seems easier to restart training.

In the previous post about when I started meditating, I talked about the strange phenomenon of feeling like I had ’two minds’ — with a secondary mind watching the first, and even passing it knowledge or information. It’s a surreal feeling, almost a high, but chasing it will get you nowhere. Try to find that feeling, and it will never return.

But meditate often enough, and it seems to come back of its own accord more frequently. Last year I found it happening more and more, and sometimes even outside of meditation practice.

It’s quite a hard feeling to explain, but I’ll try.

During meditation, one is trying to let thoughts come and go without them taking over. Without chasing a thought off down a path — which is usually what happens with thinking. In time, with enough meditation, one notices when this is happening, and it becomes easier to notice thoughts running away and to let the process ’end’ by coming back to focussing on breathing.

What I discovered happening more and more last year was a secondary process doing the noticing for me. It was as if my conscious mind didn’t need to notice a thought arising; another part of my mind did it for me, with no conscious thinking. So, a thought might arise, but then vanish almost immediately, like a cloud dissipating away.

This automatic noticing carried on in everyday life too. If something annoyed me, for example, I’d notice the annoyance arising almost immediately, and just through noticing it, it subsided and went away. If I were speaking with somebody, words would appear in my mind as if being fed to me like a script.

When this automatic noticing is happening, it can become quite an intoxicating feeling. A high. It’s almost magical. But notice it, feel elated, and it too will vanish. It’s like a cruel joke. Meditating gives you this extraordinary ability, but if you notice it, recognise it and enjoy it — it instantly goes.

I’ll persist with the 20-minute sessions this year, I think. Three or four times a week feels like the right balance for me. I have other wellness habits to fit in too — cycling, yoga; the occasional run.

Going into my tenth year of meditating, there is one thing I am sure of, though. It’s a very worthy tool to have in one’s possession. It’s not a habit I’ll ever give up.

Addendum

Interestingly, I read an article about Transcendental Meditation in the FT this weekend. There was a link in the article to this scientific research about the effects of meditation on the brain.

Thanks for reading this post via RSS! 👏

A few years ago I decided I needed to understand CSS, and built a Hugo theme using SimpleCSS. That got morphed into MVP.css, then finally, only last weekend, I swapped it out for TailwindCSS.

There are many positive aspects to using a framework to build software, but these days I think that almost everything in tech is overcomplicated, mainly because of using frameworks for everything. People have lost the art of understanding what they are building. That’s a disadvantage when things go wrong.

To satisfy my curiosity, I ended up doing a frontend web development course last year. Of the many horror show things I was introduced to (in what world is React a good idea?) Tailwind was a balancing act I could get behind.

The trouble with CSS frameworks like Bootstrap and Bulma is they’re massive. Circa 13,000 lines of CSS 😱 If one is trying to build a lean site that is far from the problems of the modern web, that simply won’t do.

So I kept the same look, but moved the highly customised MVP.css theme I had, to Tailwind.

One thing I quite liked the idea of doing was making code blocks look like a terminal. With a smidgeon of help from Claude I came up with this render hook in Hugo:

<div class="overflow-hidden rounded-sm shadow-lg">
  <!-- Mac-style window header -->
  <div class="flex items-center gap-2 bg-gray-200 px-4 py-2 dark:bg-gray-700">
    <div class="size-3 rounded-full bg-red-500"></div>
    <div class="size-3 rounded-full bg-yellow-500"></div>
    <div class="size-3 rounded-full bg-green-500"></div>
  </div>
  <!-- Code content -->
  <div class="bg-white px-8 py-2 dark:bg-gray-900">
    <pre
      class="overflow-x-auto whitespace-pre-wrap break-words"
    ><code>{{ .Inner }}</code></pre>
  </div>
</div>

Even with Tailwind, this site is still around 70kb for first load. The CSS itself is just under 1,000 lines — actually smaller than it was as bespoke CSS. I can imagine some people will argue HTML ends up looking messy with Tailwind’s utility classes; but I don’t mind that approach here. It’s Hugo templates anyway, and looking at all the ‘control’ of layouts in one file has its advantages.

Thanks for reading this post via RSS! 👏

Whilst rummaging around in a large box of cables (you’ve got one of those too, haven’t you?) the other day I found the first Apple product I ever bought, in 2003. A second-generation iPod. An object of beautiful simplicity, of design elegance not seen since Rams’ Braun of the 1980s. From the moment you unboxed an iPod, the entire experience was like no other — who, up to now, had thought to make packaging part of the journey? Amazing.

Obviously, I was curious whether it would even power up, let alone actually work. I rummaged some more and found the power block and strange cable. I’d forgotten that these early devices used FireWire! As I sat it on its charging station, there was a whirr — of a hard drive — and the iPod lit up, its display as clear as the day I’d unboxed it.

I doubt I’ve used it in over 15 years. It’s certainly been nowhere near power since we moved house almost a decade ago. Sat dormant with no charge for at least 10 years. Yet here it was, spinning up as though it hadn’t taken part in the passage of time.

Wondering if it would play, and if there was even any music still on it, I grabbed a pair of headphones. Plugging them in, I quickly remembered how the simple, delightful interface worked. There was music here. A reminder of my tastes two decades ago (somewhat more energetic than today). I chose a growly Audioslave track and hit play.

Another short whirring noise, and my ears were filled with music. It worked perfectly!

I spent the rest of the day with it in my pocket, putting tracks on now and then, wondering how long the battery would last. The answer came after about two hours.

Sitting in the same box of detritus was an old Garmin eTrex GPS. I tried that as well — it took standard AA batteries, and although it powered up, the LCD was useless. It had definitely acknowledged the passage of time, and like us humans, time had taken its toll.

My grandfather used to lament that ‘modern’ things were not built like they used to be. Those grumbles came from him whilst I sat, watching in wonder, at whatever he was crafting in his garden shed — in the early 1980s. Does every generation, as we age, have a feeling that “they don’t make them like they used to” I wonder?

The products Apple made in the late 1990s and early 2000s were truly innovative. Those early iPods were beautiful, and as a testament to their tasteful, simple design, they still are. Clearly, they also used premium components and engineered them extremely well.

Now, how do I get FireWire 400 talking to USB-C in my modern laptop, so I can sort out the music on it?

Thanks for reading this post via RSS! 👏

(Picked up via the excellent Weekly Filet newsletter)

Thanks for reading this post via RSS! 👏

I built a web application using Go, HTMx and TailwindCSS using Claude Code. I wrote zero code myself. I am not a programmer.

The longer version

I taught myself to program as a kid, in the early 1980s, on a Commodore Vic 20. Fast forward a decade and I’m a year out of university, having quit my Computer Science degree, working in my first tech job. But it’s not a coding role — I’d decided at uni that I didn’t want to be a programmer. It was technical though, and frequently involved writing ‘glue’ — shell scripts and the likes.

The next 20 years were similar — no real ‘coding’ in the sense of building applications, but lots of scripting. I taught myself countless languages to solve small problems — Perl, Python, C, Go, awk. The list goes on. But it was all rough, and because I didn’t code day in, day out, I was slow.

When I stopped being ‘hands on’ technical, around 2014, I ended up in Product Management and Product Marketing roles. In the former, I could often ask better quality questions of coders I interacted with, because at least I’d been fairly technical.

Over the last decade, I’ve had a few ideas for products, but knowing I couldn’t really build them myself, eventually the excitement would die. I noodled with some programming again when ChatGPT came out in 2022, and over the next couple of years solved a few problems by steering it, then Claude, to write code for me. A bit of Go here, a bit of Rust there.

Then a conversation with a friend a few weeks ago prompted me to try something bigger and bolder. Said friend had a fun idea to build a new social network, based on writing haikus. There was, is, something fun about the forced structure that causes one to consider one’s words more carefully. Creativity through constraints, if you like.

I jokingly said to him that he should try asking an LLM to build it for him (he’s not “technical”). Then said if he hadn’t tried it by the following week, I would.

Two weeks passed. I went on holiday. The idea kept running around my head, so I decided to see if Claude could put it together for me.

I’m no programmer, but…

I got quite frustrated with the tech industry a long time ago. On the whole, we’ve lost our way. Complexity rules, and there are so many overwrought solutions. Code is terrible — people use layers and layers, frameworks on top of frameworks, which altogether make things slower and horrendous to debug when things go wrong.

But that frustration drives me to strive for simplicity. Constraints in tech used to force it, but now compute is abundant, everybody got lazy. So I prompted Claude to work within constraints:

I’d like to create a fun ‘social network’. Fundamentally, every message has to be a haiku - so the front end needs to check the format works, and not allow a submission if it doesn’t. Given this is syllables, I don’t know how to achieve the checking. That’s your job :)

Otherwise, here’s what I expect:

Backend code is Go, using as much of the standard library as possible. If you have to use external libraries, pick the smallest (in terms of codebase) and well maintained. i.e. if there are two libraries, with one being smaller than the other, but the larger one is more recently maintained, go with the larger.

I want to minimise frontend JavaScript, so let’s use HTMx to keep it small and tight.

I’d like all CSS to be tailwindcss.

Initially, let’s use sqlite for storage, but code with it in mind that we might need to swap that out later for a proper RDBMS (which will be PostgreSQL). Don’t use an ORM - let’s just use SQL statements. Bear in mind they might need to change at some point too, so code them in such a way that it’s easy to maintain and easy to extend (think about database migrations, for example).

The first iteration worked, immediately! It compiled, and it looked great (minimalist, but great). Syllable checking was a bit hit or miss, and continues to be. But it’s easy enough to fix.

I put that first prompt into the Claude desktop app, but then thought I’d try Claude Code on it. This is when my mind was blown.

Continuing to iterate the application with CC, using just ‘English’ to make changes, I lost hours of an evening. Dare I admit, I was enjoying myself. I’d found ‘flow’. I don’t have enough experience of Go to know if it’s good code or not, so I asked another friend who can program. His verdict:

On the one hand this is miraculous for non-coders. It actually works, it’s a giant step forward from your first “Hello World!” application. Maybe that’s the way to think about it, this is the new normal for a first attempt at app development.

It’s pretty readable, easy to understand. Things are, more or less, named for human consumption.

Occasionally I’d look at the code, and if something obviously ‘smelled’ to me, I’d question Claude about it. For example, I noticed it was building custom CSS, despite me saying it should use TailwindCSS. Pointing this out, it quickly refactored everything. Right now, most LLMs seem to favour using Tailwind v3, understandably, as v4 only came out a few months ago. This also means a lot of LLM choices are flat out wrong. Luckily, Tailwind now has a sponsor program (essentially a subscription) — and as part of the initial offering, Adam has shared the markdown file he uses to steer LLMs. Adding this to Claude Code with a ‘read @file’ means it has guidance to use Tailwind as Adam intended.

The site has been running for a few weeks now. We’ve shared it with some friends to test, and it all works well. It’s running on a tiny virtual server, in a FreeBSD jail (continuing the theme of keeping things small and simple). We have plans for a ‘production’ release soon.

There’s plenty of negativity around LLMs replacing programmers. I don’t believe that. I neither think ‘AI’ is the panacea to all our woes, nor is it the worst thing in the world. As with many things, it’s more nuanced than either extreme.

For coders who think it’s nonsense and eschew AI altogether - they are cutting off their nose to spite their face¹. For the already highly skilled coders who look at using AI, they’re going to find their productivity massively sped up. They might not want, or need, to give all their coding to the AI as I have, but they could reassign some junior tasks. This is no different from using automation to scale skills. We’ve made that mistake before, and many technologists continue to make it - believing that technology will 100% replace people². In many scenarios, this produces more inefficiencies. The reality is automation is just another tool that is best coupled with human skills to produce something even better. The whole being better than the sum of the parts.

Will the likes of OpenAI³, Anthropic, et al. ‘enshittify’? Only time will tell. But I continue to believe optimistically in Anthropic’s motivation.

Thanks for reading this post via RSS! 👏

One camera that I’ve struggled to get it right with, even just holding up a slate, is the Insta360 One X2.

I’d started to look at replacing the UltraSync Blue TC generators with Deity TC-1s, as I’m also finding bluetooth simply isn’t reliable enough. I already use a Deity PR-2 mic, and it’s rock solid. It also turns out you can jam DJI Action cameras using just its USB port (and I’ve tested this works with an ageing, but still utterly reliable, DJI Action 3).

I use Final Cut Pro for editing, which is a bit of a pain with audio timecode LTC as it can’t access it. Apparently, DaVinci Resolve can. I’m not swapping my workflow for this one, niche, feature though.

Off down another rabbit hole I went.

It turns out that there is a free, open source, piece of software called LTC-tools, which is available on macOS via Homebrew. Using that, plus ffmpeg (documented in the previous post), gives me a simple workflow to add LTC as metadata timecode that FCP can read.

Because LTC-tools only works on audio files, it’s a multi-step operation to extract and embed the timecode, but it’s pretty simple once one has all the tools lined up.

For 360º video, I use Insta360’s own Insta360 Studio to turn the video into an equirectangular projection file suitable for direct import into FCP (where I’ll choose the framing later)

Then it’s a case of extracting the audio, running ltcdump against it to get timecode, and adding it via the first frame the same in the previous post.

$ ffmpeg -i VID_20250314_122622_00_001.mp4 -vn -acodec libmp3lame audio.mp3

$ ltcdump -F audio.mp3 2>/dev/null | awk '/Timecode/ {getline; getline; print $2; exit}'
12:25:22:21

I’ll probably make another Shortcut to do this in due course. For now, this is a pretty easy solution for me.

Thanks for reading this post via RSS! 👏

It will look fine if tested on opengraph.dev, but post it on LinkedIn, or test with their preview page, and it will fail.

So you can either have boring links with no rich display, else downgrade your TLS.

Thanks for reading this post via RSS! 👏

Coding

Since ChatGPT came to prevalence in late 2022, I’ve written Go, Perl, Rust, various bits of shell (including awk, sed, et al), CSS, a smidgeon of Javascript and a lot of Python/Pandas in Jupyter. All with the help of what feels like an experienced (and patient) colleague sat at my side.

It probably helps that I used to be technical, so can steer an LLM to generate succinct code. I will often instruct it to try to stick to built in libraries, for example, to minimise code footprints. If an external module is required, and there are choices, I will grill the chatbot on which one is the smallest. Small is beautiful¹; simplify and add lightness², and all that.

Along with using ChatGPT, I’ve experimented with some local LLMs running under Ollama (interesting aside: I found a 2017 PC with an i7 chip and an Nvidia 3080, is twice as fast for local LLMs as an M1 Max MacBook Pro) and had occasionally tried Anthropic’s Claude.

I tend to run everything inside of TypingMind’s excellent UI (although also occasionally use Charm’s mods on a FreeBSD server) — which is great for switching between multiple models. Also, both OpenAI and Anthropic’s APIs are far cheaper than a monthly app subscription.

Pause for thought

Recently, three things coincidentally happened that have made me gravitate towards Claude as my general purpose LLM of choice.

1. Coding with Claude has produced better results
2. I listened to an interview on Lenny’s Podcast with Seth Godin
3. I listened to an interview with Amanda Askell by Lex Fridman

They happened in that order, over the space of about five days. Let me break down how they came to influence my thoughts on Claude.

Tech choices

I’d been trying to ‘port’ a complex Excel spreadsheet to Jupyter for a few months. ChatGPT had failed multiple times to get it working, and my own coding skills were, as per usual, letting me down. I gave it to Claude Sonnet, and over the space of 3 or 4 hours, completely cracked it. What I found especially useful, besides it actually working, was Claude’s helpful suggestions along the way. It wanted to indulge further thought and explanation (yes, the cynic in you might well say ‘obviously, because that generates more revenue with API use’, but let’s gloss over the cynicism for now. I’m a pragmatic optimist) and would often proffer valuable next avenues to explore. Even the free Claude app has delivered a few quick code solutions for me.

Seth Godin

Then I was listening to Lenny Rachitsky interview marketing legend, Seth Godin. Seth talks about Claude in this portion (link should take you to around 18 minutes in), which really resonated with me, as I’ve long held the thought that a company’s culture is reflected in its products (bad culture = bad product. Tangentially, I had it in mind to write a blog about that some years ago, but then discovered a succinct piece in Don Norman’s excellent book, The Design of Everyday Things). Lenny also mentioned the cheeky ads Anthropic are running.

Amanda Askell

Despite what Reddit may say, the decisive point for me came in the second interview I listened to this week. I don’t usually listen to Lex’s interviews, and when the YouTube algorithm put the interview with Dario Amodei on my feed, it caught my interest. Straight away though, I was turned off by its 5 hour run time. Ridiculous! I don’t have time for that. But, as I often like to check my initial gut response, I took a look at the timestamps in the description, and saw that it was actually three interviews. I’d not heard of the other two Anthropic staffers interviewed, but the intro piece for Amanda Askell piqued my curiosity further:

You are a philosopher by training…

I ended up listening to the full interview with her, as she is largely accepted as the personality behind Claude. Once you hear her speak and listen to her thoughts, Seth’s comment starts to make sense in the context of your product is a result of your culture. Even if I hadn’t heard Seth, or even discovered Claude Sonnet’s ability to code better³, the interview with Askell probably would’ve created a sense of ‘brand loyalty’ in me.

I’ve mentioned anthropomorphising AI in a previous post, and here, at around 3h11m, Amanda talks about over anthropomorphising, and interestingly, under anthropomorphising, when conversing with an LLM chatbot.

Kagi

There’s no doubt that general purpose search engines are more or less useless for finding information now. I’ve been paying for Kagi for most of this year, and use it as my daily driver search engine⁴. Kagi uses machine learning, and its search results are superior to anything Google, DuckDuckGo et al provide (Kagi’s motivation isn’t to sell adverts, unlike other search engines). I often cross reference things gleaned in Claude with Kagi. I find that gives me a good way to keep the LLM in check (I have noticed Claude occasionally start out by saying it’s just an LLM and can ‘hallucinate’, so to check its answers. If only more humans had the will to self reflect).

Do you really need my phone number?

I’ll finish with a slight grumble — in that I found having to use a mobile number to sign up for Anthropic overly intrusive. Really, there’s no need to have my email and mobile just to use the service. I’m starting to give up on lots of services that gather too much personal information, as most companies are sadly proving they can’t be trusted with user data. If Anthropic hadn’t asked for a phone number to register for Claude, my wife would’ve moved from ChatGPT. As it is, they lost a potential user. I wonder how many more people get turned off by that?

There is a lot of negative press about LLMs, but the genie is out of the bottle now. The naysayers may suggest AI will go no further and die, but I don’t believe that’s going to happen. There is genuine utility in AI use and in the right contexts, it can be a massive lever for metaphorical human lifting.

Additional updates

As I find additional, useful, information I’ll continue to update this post.

Ben Evans linked to a Bloomberg interview with Dario Amodei in his newsletter in May 2025. It’s a good read, and in true confirmation bias fashion, it reinforces my hunch that Anthropic are trying to do the right thing. John Gruber also linked to interesting information about the Claude 4 testing.

Rick Rubin’s interview with Jack Clark is another interesting chat with a co-founder of Anthropic that continues to reinforce my belief they are thinking about AI in sensible, pragmatic ways.

Last week (22nd Jan, 2026) Anthropic released a blog about Claude’s new constitution. There is also confirmation that Amanda Askell is indeed the lead behind Claude’s ‘character’. Incidentally, there’s a new interview with Askell on Anthropic’s YouTube channel.

Thanks for reading this post via RSS! 👏

Hugo usefully has RSS feeds built in and I’ve long included an RSS version of this site. However, the default template only lists part of the article for reading in an RSS reader.

Whilst perusing Kev Quirk’s blog via NetNewsWire recently, I noticed he has a footer message thanking the reader for looking at the article via RSS.

I thought this was a neat idea — rather than forcing people to click away from the reader to the full site (“to what purpose?”), just let them read the whole article wherever they’re reading from.

With some inspiration from this post, I made a custom rss.xml file in my Hugo template layouts folder and changed the description bit to this:

  <description>
  {{ with .Content }}
  {{ printf `<![CDATA[%s]]>` . | safeHTML }}
  {{ end }}
  {{ `<![CDATA[<hr><p>Thanks for reading this post via RSS! 👏</p>]]>` | safeHTML}}
  {{ printf `<![CDATA[<a href="mailto:%s?subject=%s">Reply to this post by email</a>]]>` $authorEmail .Title | safeHTML }}
  </description>

This expects $authorEmail to be set in params: of your Hugo config (hugo.yml in my case):

params:
emails:
  rss: "your@email.address"

Now when reading anything on this blog via RSS, you’ll get the full post right there in your reader.

Thanks for reading this post via RSS! 👏

Things by Cultured Code is the best app I own. And I use the word ‘own’ quite intentionally.

During a discussion with a good friend a few days ago, he mentioned he organises his todos using the Things app. We’ve been friends for almost 20 years, and I’m surprised we’ve never discussed this before, truth be told. Because I’ve been using Things since 2008! (our conversation caused me to look through its logbook, as I couldn’t remember when I started using it.)

Near the beginning of my tech career, 30 years ago, a wise man told me I had to write things down that needed doing. Else I’d forget. He was probably right (it still astonishes me today when I find somebody who doesn’t note todos, whether old skool analogue or digitally).

I went through many notebooks and ‘systems’ (think: graph paper, tick boxes, lines, codes, etc.) before discovering Things. I like admin to be simple, and most tech applications are anything but (design, the problem is design; or rather, a lack of good design).

Things, however, is both simple in its interface, and in its user experience. Over the course of 17 years of use, the developers have resisted the temptation to mess with it too much, keeping it simple. Design right out of Dieter Rams’ book.

It’s also blazingly fast to synchronise notes. I’m constantly amazed, with all the crud mine has accumulated over the years, that it’s still so fast.

And the most important detail is, Things is still a one time purchase. No subscriptions! No enshittification.

Thanks for reading this post via RSS! 👏

When Apple Maps was first released it was terrible. Over the following years it steadily got better, but was mostly still inferior to Google Maps. The gap is now pretty small though. Enough to start using it over Google maps?

On a recent road trip holiday (the north of England to Sweden, 2,800 miles, since you ask) I decided to try and do all navigation with Apple Maps rather than Waze or Google.

If a route ended up deep into a city, I’d compare notes with Waze, and pretty much Apple Maps would always mirror the same route. So I stuck with it. And I’ll admit, it never let us down.

Apple Maps has long since had a stunning 3d view, that shadows anything the default Google Maps can offer (Earth is a different story, but we’ll ignore that for now as it’s not so readily accessible), and I find Apple’s street view far clearer in its indication of which way you’re facing:

Compared to Google Maps’ little arrow:

I tend to couple it up with what3words (which I’m a massive fan of) for the final destination — finding where I want to go with w3w, and sending it to Apple Maps for navigation. This works well for navigating directly to, say, a hotel car park entrance that may not be the same spot as the hotel address.

The only negative left, I’ve found, is searching for a type of business versus a name. For example, searching for ‘specialty coffee shop’ on Google Maps will find you really good coffee shops. Try that on Apple Maps and it will look for a business called “Specialty Coffee” 🙄

Still, that’s a small negative to have to jump back into Google Maps for. And it’s yet another nail being hammered into Google’s coffin. No bad thing.

Thanks for reading this post via RSS! 👏

1Password is a good password manager, relatively speaking. But I’ve got rid of it, and this is why.

I’ve lost track of the years I’ve been using 1Password for. It was a great password manager, and there’s been little controversy attached to it, unlike the competition. Managing passwords in a central place always felt like anathema to me, but needs must, and my former solution — a gpg encrypted text file 😂 — wasn’t going to cut it with the wife.

Their original implementation, of an encrypted file simply shared via iCloud, felt less yucky than a subscription cloud service, and I was one of the many who grumbled when they forced users down their own cloud storage route.

Since then, the service has steadily worsened as it’s grown with lots of needless functionality. And there’s an obvious reason why.

VC-induced enshittifcation.

$620 million! Good grief. How they’ve managed to sell the idea that managing passwords is worth 6.8B is beyond me.

What scares me most about that level of investment though, is the VCs will put metaphorical thumb screws on the business at some point. It always happens, without fail.¹ With that pressure will come mistakes. So as much as I’ve trusted 1Password to date, and there is no evidence to say they will cock up, I’m not willing to take the chance.

With the breaking out of Apple’s own password store — keychain — to a separate app, and family sharing functionality, it’s finally a good enough solution to just let the OS be a password manager.

If you want to move away, and just use Apple’s password manager, Jamie Zawinski has a post about migrating. It’s not hard. I chose to overlap my use for most of this year, simply saving passwords as I used them. It’s been a great way to clear out a lot of dead wood.

Thanks for reading this post via RSS! 👏

You’ve most likely heard the phrase “Jack of all trades, master of none”. It’s most often used as an insult, or at least in a derogatory context. But that’s not the full version of the phrase. And the insult isn’t in the direction you think.

ONE OF THE EARLIEST recorded uses of the phrase “jack of all trades” as an insult dates to 1592. In the New Latin form “Johannes factotum”, it was contained in a pamphlet by a playwright criticizing his own industry. The jab refers to a poet with no university education who was apparently involved in various other roles, like copying scripts and bit-part acting, even trying to write plays. The poet on the receiving end of the insult: a young William Shakespeare.

The phrase evolved over time, and today it’s usually “jack of all trades, master of none.”

I think it is culturally telling that we habitually hack off the end of the long version:

“A jack of all trades is a master of none, but oftentimes better than a master of one.”

Quote from David Epstein’s “Range”.

Thanks for reading this post via RSS! 👏

I used to quite enjoy writing regexs. man perlre is one’s friend. However, sometimes you have to realise there’s a quicker way.

This week, a good friend and I have been having a lengthy chat, over texts, spanning a few days. We’ve been discussing the unfortunate ‘dumbing down’ of the tech industry, and how ’environment determines behaviour’¹ — the more abstracted and layered in frameworks things become, the less engineers actually understand what’s happening underneath it all. The less they understand, the less they seemed to want to understand. And so skills are vanishing.

So with a massive dollop of irony, I asked ChatGPT to do something for me that ordinarily I’d a) do with relative aplomb b) probably enjoy.

For a while now I’ve had a blocklist on my web server configuration that blocks known AI bot scrapers. I convert the robots.txt list into a configuration line for nginx, as per Robb Knight’s blog post.

Every time the list gets updated, I change the string in nginx. But today I couldn’t be bothered to work out what had changed, and just wanted to copy & paste the list. I was about to put the list into Vim and record a quick macro, when it dawned on me I should probably write a script and submit it to the repo. But then I thought, “I wonder if ChatGPT can sort this quicker?”

Answer, yes it can:

Hey! Can you take this list and remove "User-agent:\s" then join every entry together with | please?
...
Here’s the list with “User-agent: “ removed and all entries joined together with |:

Finding this irony a touch comical, I posted it to the discussion area of the GitHub repo. That conversation prompted a response, and a question that caused me to further dumb myself down:

https://github.com/orgs/ai-robots-txt/discussions/34

So there we go. A one-liner awk is the result:

awk '{gsub(/^User-agent: /, ""); s = s $0 "|"} END {sub(/\|$/, "", s); print s}' user_agents.txt

Thanks for reading this post via RSS! 👏

Something a little bit different to the usual things I write about here. This post is an aide-mémoire for me as much as anything, but maybe you’ll find it useful too. I’ve recently been trying to utilise timecode on a number of video and audio devices to help with editing in post. After a few false starts, I think I now have the tools, and a process, that works.

For the last year I’ve turned a hobby into useful—and dare I admit, fun—work. Towards the end of 2022, I decided I needed a break from full time tech-industry work, and happened to mention this to a local garage that serviced my car. Cue a short conversation, and me agreeing to help them build a YouTube channel. It was a great opportunity to make use of the knowledge and experience I’d built up of marketing and communities in software companies. The goal was not a monetised YT channel, but a ‘marketing funnel’ to raise awareness of a young brand.

Complexity

As time has gone on, our video ideas have become more complex. After a recent filming trip, I found myself trying to coordinate footage from 10 cameras and three sources of audio!

I already owned Zoom audio equipment, and researching timecode use for them led me to Timecode Systems’ (now owned by Atomos) UltraSync products, in particular their Blue device.

A very helpful friend already owned a couple of Blues, so I borrowed them to try out. Syncing the audio devices with my main camera was easy enough, especially since Fujifilm have recently added UltraSync Blue support to the X-H2S.

The problems arise with devices like DJI drones and older Osmo Action cameras (the newer 3 and 4 support timecode jamming via a cable.

Adding timecode with ffmpeg

Whilst down another rabbit hole on using ffmpeg to re-encode ProRes footage, I discovered you can embed timecode; and the UltraSync Blue app is also a slate.

The easiest solution for these older devices seems to be relatively simple then — start recording facing the UltraSync Blue Slate app to grab the timecode that instant, then use ffmpeg to embed timecode in the clip metadata.

That’s an important thing to note — you need the timecode right at the beginning of the clip. So don’t start recording, then put the slate in front of the camera. Timecode embeds from the absolute beginning of the file, so it’s important to have the image as the first frame. Or, at least, trim the clip so it’s the first frame (but that’s another step, ergo faff).

From the first frame, we now know what the starting timecode is and it can be embedded in the video file. I found a relatively cheap app to help with this, QTchange. Under the hood, it uses ffmpeg to do the work, and since I’d only recently built an Apple Silicon specific version of ffmpeg (good for moving video processing off to the embedded GPU in my laptop’s M1 Max processor) and was comfortable working in a terminal, I didn’t see the need for it. If you prefer using graphical interface tools, then QTchange is probably a worthwhile spend.

With the image above as an example, we know the starting timecode is 10:28:22:00. To embed that in a video file we can use these switches with ffmpeg:

$ ffmpeg -i dji_mimo_20240602_102956_23_1717320826054_video.mp4 -timecode 10:28:22:00 -codec copy osmo.mp4

Note that -i is the input file.

Now when I build a Multicam Clip in FCP, every clip has timecode embedded, and they synchronise perfectly:

A relatively cheap solution, that will save hours of work.

Update, October 2024: I’ve now automated most of this with Apple Shortcuts. I’m sure these two could be folded down into one shortcut, but I was developing them in separate steps, and now it works to the point I can’t be bothered to distil it further. One shortcut extracts the first frame and OCRs the numbers (timecode) out, then a second shortcut applies the timecode. Here’s what they look like:

Thanks for reading this post via RSS! 👏

Along with ‘Thinking, Fast and Slow’, I’d recommend How Minds Change as essential reading. The foundations of human behaviour understanding come from Kahneman’s book, and here McRaney builds on it with real world examples. Through meeting various interesting people, McRaney paints a picture that tells a simple truth:

For one thing, from writing my previous books, I knew the idea that facts alone could make everyone see things the same way was a venerable misconception.

I made 63 highlights in this book. There are many small gems that can serve as useful guides for better thinking. Here, I’ve picked out a few of those highlights to discuss.

reality as we experience it isn’t a perfect one-to-one account of the world around us. The world, as you experience it, is a simulation running inside your skull, a waking dream. We each live in a virtual landscape of perpetual imagination and self-generated illusion, a hallucination informed over our lifetimes by our senses and thoughts about them, updated continuously as we bring in new experiences and think new thoughts about what we have sensed

It’s quite profound to realise there is no single, true, reality. What we think of as reality is just perception. And our perception of the world is context sensitive. That context is made up of years of experience and the situation we find ourselves in at the time. Have you ever held back on an opinion, thinking it clashes with a peer group at that moment? Not spoken up in a meeting, because everyone around you thinks something is a great idea, when you feel it’s anything but? Context, shaping your reality.

A fine example of our reality being shaped by context, is The Dress. Do you remember that bizarre phenomenon, and how you could easily argue with somebody stood next to you that it was white and gold, when they were 100% sure it was black and blue?

when the truth is uncertain, our brains resolve that uncertainty without our knowledge by creating the most likely reality they can imagine based on our prior experiences.

In Thinking, Fast and Slow, Kahneman’s work showed that our — emotive, reactionary — System 1 has a reality it knows, and when it comes to defending that reality, we’ll simply draw on the experience we have. This is why we tend towards confirmation bias.

It’s useful to think of confirmation bias as the goggles we put on when we feel highly motivated by fear, anxiety, anger, and so on. In these states we begin looking for confirmation that the emotions we feel are justified.

I once read an interesting take on what happens when one tells somebody ’they are wrong’. The uncomfortable sensation they feel, is down to their reality being smashed in that moment. It’s a shock to System 1 — and the fight between it, its reality, and the new information System 2 has received, begins. Cognitive dissonance.

If you add any conflict over resources of any kind, humans will instinctively enter us-versus-them thinking, even if that’s not the overall most beneficial strategy.

‘Conflict over resources’ can render itself as a disagreement, or the urge to be right, come what may. That’s a whole other discussion, principally around ego, but we’ll leave that for now and recognise something obvious about the way our brains default to tribalism.

Homo Sapiens have existed for around 300,000 years¹, yet it’s only for the last 100 years or less that we’ve not had to mostly focus on survival. To put that into context, baked into our brains are instincts that have been in control of our behaviour for over 99.96% of our species’ existence. Those instincts don’t always fit neatly into the modern world, but they do find parallels to relate to.

Most of the time we see the world as we expect to see it, and most of the time that’s fine; but the brain often gets things wrong because it prefers to sacrifice accuracy for speed.

The ’laziness’ of falling back to System 1 (our ‘gut’ reaction) has an economical reason. Conscious thought — System 2 — takes more energy². It can literally be tiring trying to question our System 1.

Throughout the book McRaney meets various people whose job it is to “change people’s minds”. From their experience, he offers some guidance.

No matter the message, face-to-face messaging is far and away the most effective channel. We are biologically hardwired to respond to the human face.

Maybe this explains the growth of video-based marketing³ and the success of YouTube (the world’s second largest search engine, the largest being its parent, Google).

A few years ago I ran a fully-remote Product team. If I was asked a question on our internal Slack, I would often reply with a short video. I chose to respond this way as I felt it would present the information clearer, and with the intent I wanted. As Jason Fried recently quoted in a blog post:

anytime there are two people conversing, there are actually six people in the conversation:
Who you think you are
Who you think the other person is
Who you think the other person thinks you are
Who the other person thinks they are
Who the other person thinks you are
Who the other person thinks you think they are.

When we read words we apply our own emotion to those words. If we think the author is angry with us, we’ll read it angrily, even if that’s not the way it was intended at all. I found a message carried far more clarity of intention when delivered by video, especially when many of the recipients didn’t share a common language or have English as their native tongue. In this modern day of geographically dispersed, remote working, teams, asynchronous communication leaves far more opportunity for ‘Chinese Whispers’. Short form video goes some way to alleviating misinterpretation.

Very early in the book, McRaney explains its purpose, lest you think you’re about to learn devious techniques for coercing people to do thy bidding:

Persuasion is not coercion, and it is also not an attempt to defeat your intellectual opponent with facts or moral superiority, nor is it a debate with a winner or a loser.

This, once again, comes down to ego, and that’s not the topic of the book, nor of this blog post. As an introvert, I’m not driven to ‘win’ arguments, I seek truth. Understanding how our mind works is useful for seeking truths versus winning arguments. Everyone has their own reality, remember. The ability to understand that, and empathise, will always render kinder discussions.

[he] later said to think of questions as keys on a giant ring. If you keep asking and listening, he told the crowd, one of those keys was bound to unlock the door to a personal experience related to the topic. It was there, and only there, he said, that a single conversation could change someone’s mind.

This was an interesting chapter about ‘Deep Canvassing’, an area where the canvasser is trying to get somebody to change their mind on a topic in a short space of time. Usually the topic is highly emotive too, so initial reactions are strong. But this is a key takeaway from the book — changing minds isn’t about bludgeoning people with information. It’s about finding common ground, or helping a mind to absorb new information and draw alternative conclusions in its own time.

Once people see where their ideas come from, they become aware that they come from somewhere. They can then ask themselves if they’ve learned anything new in the time since they last considered them. Maybe those ideas need updating in some way.

Challenging another person’s reality will initially be discomforting for them. Attempting to bludgeon with facts, to get them to see your point of view, will have the opposite effect. They’ll dig in to their reality, seek confirmation bias, and appear to try to change your mind (that’s not what they’re actually doing, really they’re defending their reality). This was discussed in Thinking, Fast and Slow, and I’ve mentioned it previously.

Sometimes a discussion in the moment won’t yield the result you desire. Give people time to introspect though, and you may find that they change their mind. Trying to force the point will cause them to remain overconfident in their understanding of the topic they’re passionate about, and overconfidence becomes certainty in our heads.

Ask questions. Plant seeds. Give them time to grow.

Thanks for reading this post via RSS! 👏

I may have ‘retired’ from being a techie a long time ago, but I haven’t stopped solving little day-to-day problems in a shell 😊 So like any self-respecting nerd, whether old or young, one needs a good prompt. It’s a pity that Starship’s setup on macOS seems to fail out of the box though. Here’s how to fix it.

Starship is a fast, configurable, prompt for most of the common shells. It’s a binary, built in Rust. Compared to the popular Oh My Zsh, it’s tiny, simple and fast. Just the way I prefer my tech.

But the instructions for configuring in zsh fail on macOS, with the prompt never appearing. It didn’t really make sense, either, as a common eval $(blah) should work just fine. Searching only throws up a Stackoverflow article that doesn’t actually solve the problem.

Rummaging through the closed issues on Github turned up the answer though; quote the eval, et voila! Working prompt.

Change eval $(starship init zsh) to eval "$(starship init zsh)" and it works.

Update: It looks like the instructions have been updated on the homepage now.

Thanks for reading this post via RSS! 👏

I decided to try and save some space in my iCloud drive. Looking at the usage, 160gb was being taken up by Photos. I thought that was ridiculous. This turned out to be nowhere near as ridiculous as the ensuing saga to try and delete some files.

Just delete the files

Naturally I started by deleting every single video in the library. Photos at least makes it easy to find particular types of file, with the ‘Media Types’ album link on the left.

Next came the realisation that I had hundreds of photos that were JPEG+RAW, with the RAW files being many times the size of the JPEGs. I wondered if you could separate the two, deleting just the RAW files. A bit of searching revealed it wasn’t that easy — you have to delete the JPEG+RAW combo. So I exported all the photos, using the “Export unmodified originals…” menu option, and deleted the actual RAW files on the file system. Then simply re-imported the JPEGs back into Photos.

I thought that would be the end of it.

They’re back!

Later that day I noticed on my iPhone that the RAW ‘Media Type’ still existed — and couldn’t quite believe it when I saw that the JPEGs I’d re-imported earlier appeared to be back attached to the RAW files I thought I’d deleted! WTF?!

So I contacted Apple Support. Naturally the chat bot was no help, so I took a phone call. The very polite chap on the phone went through all the steps with me, again, watching what happened. Oddly, the RAW files would reappear first on iOS, so we wondered if there was something odd going on there — and opted to remove my iPad and iPhone from iCloud Photos, along with an Apple TV that had access too. I deleted all the files again, checked in the web version of iCloud, looked into the Photos Library; all gone. Emptied the ‘Recently Deleted’ folder in Photos. No RAWs anywhere. So the support guy signed off and left me to it, saying I could reopen the case if this hadn’t solved it (note to self: Apple’s support path is terrible. You just start from scratch again).

Dragged the JPEGs back in.

The RAWs started reappearing… 😱

If at first you don’t succeed

Not one to accept defeat, I’d thought I’d give it one last, possibly drastic, attempt. Sigh.

By this point, the only device I had sync’ing with iCloud Photos was the Mac. So I exported the files. Deleted them in Photos. Emptied ‘Recently Deleted’. Deleted the RAWs from the export folder. Emptied the trash. Checked in the Photos Library — all gone. Then made the drastic move. Turned off iCloud Photos on the Mac, and deleted every single photo in the web interface.

I now had no iCloud Photos library anywhere.

It had started to occur to me that despite iCloud drive having no history, I wondered if Apple keeps the files ‘just in case’. Maybe there’s a legal reason that they don’t talk about (I’m not going through those T&Cs to find that clause; it might well be documented, I don’t know). Anyway, I thought I’d give it a few days, then turn iCloud Photos back on to see what would happen.

Which I did, this evening, January 30th. Only on the Mac, nowhere else. There were no photos on iCloud web, the iPad, iPhone and Apple TV were all ‘disconnected’ from iCloud Photos.

The RAW files came back…

Doing an ‘Export unmodified original’ does indeed pull two files out on to the local filesystem, so the raw files have definitely come back from iCloud. Despite being deleted. Umpteen times.

All the iCloud shared albums I had also recreated themselves, but with no photos in. Which leads me to believe the photos are deleted, but clearly some ‘pointer’ is not.

I’m at a loss now. I’ve no idea where they come from or how to delete them 😂 It’s infuriating. It also begs some questions about the trustworthiness of Apple. But that’s a pondering for another time. It would seem though, that I am not alone.

Final thought

OK, I’ve turned iCloud Photos off again.

I’ll give it the 30 days, then turn the library sync back on (without RAWs in!) and see if that finally de-links them. If it doesn’t, I might just have to scream.

Update: 7th March, 2024

I apprehensively turned iCloud Photos back on yesterday. I’d kept a complete library on my Mac, except with all RAWs removed, of course. It sync’d fine, and no RAWs reappeared. Although, oddly, all the iCloud Photos Shared Albums recreated and repopulated with the same photos — which suggests to me, even though Apple deletes all photos in iCloud after 30 days, there’s some sort of metadata that they do not remove.

Since no RAWs had reappeared on the Mac, I braved the iPad and iPhone. All good! No RAWs!

So there we go. A painful exercise, and one that needs at least 30 days to resolve the ‘problem’. If you want to decouple RAW photos from JPGs in iCloud Photos, you need to disable iCloud Photos on every device, export the photos and RAWs from your library, delete them, reimport the JPGs, and wait at least 30 days before turning iCloud Photos back on. What a total faff.

Thanks for reading this post via RSS! 👏

Taking screenshots on macOS is a doddle. cmd-shift-4, or cmd-shift-3, or for ultimate control, cmd-shift-5. What could be simpler? Why would you want more? That’s what I used to think, until I tried CleanShot.

I’ve often found myself wanting to get a screen grab of, say, a browser window, but in a particular ratio. For the last year I’ve been running marketing for a local business. As part of that, I create a lot of content for YouTube. So quite often I want a screenshot in 16 x 9 ratio, which is awkward to do. I’ve either ended up writing Apple Script to size the window, then grabbed it with cmd-shift-4, or grabbed the whole screen then cropped it in Pixelmator. What a faff.

Over Christmas, whilst joining in the default apps trend and viewing others’ apps, I saw somebody using an add on app for screen capture — CleanShot X. My immediate thought was ‘why would you need that, the default suffices’. Then I tried it.

I’m now hooked. The one-time purchase price is worth the time and effort savings (I’m not fussed about the subscription version — in fact, most software subscriptions are a turn off these days).

What value am I getting from it? Well, fixed ratio grabs for starters. But neat extras like being able to add a background to a grab is useful (like the image at the head of this post). CleanShot just makes everything easy, and even ‘hijacks’ the macOS standard keyboard shortcuts.

Like Bartender, it’s one of those little helpers that actually makes the experience of macOS even better.

Thanks for reading this post via RSS! 👏

📨 Mail Service: Hey
📮 Mail Client: Hey
📝 Notes: Apple Notes (+ Bear)
✅ To-Do: Things (+ Apple Reminders)
📆 Calendar: Fantastical — fronting Apple iCloud
📆 Calendar bookings: Cal.com
🙍 Contacts: Apple Contacts
🗞️ RSS Client: NetNewsWire
⌨️ Launcher: Spotlight
📸 iPhone photo shooting: Halide
☁️ Cloud storage: iCloud
🌅 Photo library: Photos (iCloud + Glass]
🌐 Web Browser: Safari
💬 Chat: iMessages (+ Signal and Discord)
📺 Video meetings: Whereby
🔖 Bookmarks: Safari + Notes
📚 Reading: Kindle
📜 Word Processing: Apple Pages
📈 Spreadsheets: Apple Numbers
📊 Presentations: Apple Keynote
🛒 Shopping Lists: Things
💰 Personal Finance: Apple Numbers
🎵 Music: Apple Music
🎤 Podcasts: Apple Podcasts
🔐 Password Management: 🤐
🤦 Social Media: Mastodon
🌤️ Weather: Windy, Openweather, Metoffice
🔎 Search: DuckDuckGo
🧮 Code Editor: Mostly Vim (+ Nova)

Thanks for reading this post via RSS! 👏

Elaine Moore recently wrote an article in the FT titled “We might be surprised by our reactions to generative AI”. It got me thinking about the way we ‘speak’ to generative AI, like ChatGPT.

Like many of the tech community, I’ve been using ChatGPT for most of this year. I’ve found it to be the search engine I wish we’d always had, and a fabulous co-pilot in so many tasks. As a distant former ‘hands on’ technologist, I’m very rusty in anything that involves writing code. Yet I’ve found myself wanting to automate problems away, or solve things in a simpler fashion (i.e. get a computer to do it). So I’ve turned to ChatGPT to help me write code, and I’ve found it incredibly helpful. It is a productivity booster for me.

I started out making my prompts terse, ‘search engine style’. But then I changed, commununicating with it like I would a person. This might seem a bit daft, but actually it feels better to me. Internally, the way I think about communicating with people is to be respectful and kind. I treat people the way I like to be treated.

I forget where I read it, maybe something by Kahneman or Thaler, but the quote “people’s behaviour is a product of the system they are part of”¹ comes to mind. To that end, I see being kind on the inside as a precursor to being kind on the outside. Call me soft, but I anthropomorphise my dog with the greatest of ease, and so I feel like doing the same to an AI detracts nothing from how I think about communication.

Here’s a snippet of a ‘conversation’ I had with ChatGPT about analysing the stock market. Am I daft to use manners? What do you think? I welcome your thoughts via Mastodon or LinkedIn.

Update

This sparked a bit of conversation on LinkedIn, with Stuart asking if I’d tried asking AI itself 😊 Fair point. So I did:

Thanks for reading this post via RSS! 👏

After my desk recently featured on Workspaces, its curator, Ryan, mentioned that several people had asked about the teleprompter set up I use for video meetings.

I’ve been working from home for 10 years, so video meetings were nothing new to me when the pandemic started. Even by that point, I’d been experimenting with higher quality meeting set ups, using a better microphone and my everyday camera (a Fuji X-E4) as a webcam.

There’s no escaping the fact that trying to look at folks on a screen means you’re not looking them in the eye. This creates an odd disconnect not present in any normal human interaction. I have a theory that this disconnect, this lack of genuine connection (eye to eye), contributes heavily to ‘Zoom fatigue’.

So I thought I’d try using a teleprompter to put video meetings on. When the other person is full screen, this feels a lot closer to speaking ‘face to face’. The other person has the added normality of me looking them in the eye when we’re speaking, too. I feel like this goes some way to making a video call feel that bit more realistic.

I bought a large-ish teleprompter from Amazon and put the X-E4 in it — positioning the camera so that I’m looking directly into the lens when the other person’s eyes are at its level. This creates an obvious problem you’ll have considered — the image displayed is mirrored, i.e. back to front.

When I first tried this set up, I was using an iPad as the display, and had to do a lot of faffing around with OBS to flip the image. This quickly got more tiresome than the actual meetings. Then I discovered the Lilliput 10" field monitor on Amazon (via Aaron Parecki), which has a built in ‘image flip’ function.

I connect the MacBook to my desk monitor via USB-C, and the Lilliput is connected via the HDMI interface. This becomes another monitor — making it easy to drag Zoom, or whatever video meeting I’m in, to the Lilliput. The monitor flips the image, so moving the mouse around is perfectly natural.

This may not be the cheapest way to ease Zoom fatigue, (how about not having so many meetings? 🤷‍♂️) but since working-from-home saves on commuting costs, I consider it a worthwhile outlay.

Here’s a video I made a while ago about the desk setup. The only things that have changed since I made this, are the introduction of the teleprompter so I’m not looking at the camera above the computer, and the iMac being replaced with the separate screen and laptop.

Thanks for reading this post via RSS! 👏

That doesn’t mean I’ve spent my life feeling short-changed though. On the contrary, I’ve spent the rest of my life continuously learning. So maybe you’ll forgive me for having only recently discovered Max Ehrmann’s Desiderata poem. Shock! Horror! I know. It feels like it channels the core of stoicism, another rabbit hole I’ve been down in recent years (off the back of discovering mindfulness).

Here it is in full, for your enjoyment and mine…

Go placidly amid the noise and the haste, and remember what peace there may be in silence. As far as possible, without surrender, be on good terms with all persons.

Speak your truth quietly and clearly; and listen to others, even to the dull and the ignorant; they too have their story.

Avoid loud and aggressive persons; they are vexatious to the spirit. If you compare yourself with others, you may become vain or bitter, for always there will be greater and lesser persons than yourself.

Enjoy your achievements as well as your plans. Keep interested in your own career, however humble; it is a real possession in the changing fortunes of time.

Exercise caution in your business affairs, for the world is full of trickery. But let this not blind you to what virtue there is; many persons strive for high ideals, and everywhere life is full of heroism.

Be yourself. Especially do not feign affection. Neither be cynical about love; for in the face of all aridity and disenchantment, it is as perennial as the grass.

Take kindly the counsel of the years, gracefully surrendering the things of youth.

Nurture strength of spirit to shield you in sudden misfortune. But do not distress yourself with dark imaginings. Many fears are born of fatigue and loneliness.

Beyond a wholesome discipline, be gentle with yourself. You are a child of the universe no less than the trees and the stars; you have a right to be here.

And whether or not it is clear to you, no doubt the universe is unfolding as it should. Therefore be at peace with God, whatever you conceive Him to be. And whatever your labors and aspirations, in the noisy confusion of life, keep peace in your soul. With all its sham, drudgery and broken dreams, it is still a beautiful world. Be cheerful. Strive to be happy.

Max Ehrmann, 1927

Thanks for reading this post via RSS! 👏

“Indifference towards people and the reality in which they live is actually the one and only cardinal sin in design”. Dieter Rams.

When I was growing up I had every intention of becoming a car designer. Something derailed that in my teens, just as I found out I could ‘work’ computers. My career path changed, but my interest in design as a discipline never left me.

After nearly 30 years of working in tech, I do sometimes despair of the industry tendency to ignore people. Sometimes it feels like we’re making software just for the sake of making it, not recognising that we’re actually trying to solve problems for people. Although I believe most of the indifference is not deliberate, I do think it is pervasive.

A few years back I was going to write a blog titled “Your products are a result of your company culture”. When I came across this quote in the book, it made me smile:

“Producing a good product requires a lot more than good technical skills: it requires a harmonious, smoothly functioning, cooperative and respectful organisation”.

I marked a lot of Don Norman’s book whilst perusing it.

It was interesting to read his opening stories about failings in door design (admit it, how many times have you gone to pull on a glass door with a handle in a modern building, only to realise you need to push to open it?) Chapter five is, encouragingly, titled “Human error? No, bad design”.

“Design is concerned with how things work, how they are controlled, and the nature of the interaction between people and technology. When done well, the results are brilliant, pleasurable products. When done badly, the products are unusable”

The early part of the book covers human psychology, and it was pleasing to see the overlap of design and almost every other facet of life. As you’d expect, given we’re all still just people. When designing any form of product, it pays to be cognisant of human behaviour and not just use logical thought to derive the solution. Here’s a neat example of design thinking:

It’s amazing how simple the ideas are in that video. The continuous colour of the pavements, that their levels create the boundary focus, i.e. on people walking and cycling.

Getting back to DOET…

“Because we are only aware of the reflective level of conscious processing, we tend to believe that all human thought is conscious. But it isn’t. We also tend to believe that thought can be separated from emotion. This is also false.”

Norman covers memory models, knowledge and even how culture can affect the way we map knowledge and experience. It’s important to recognise that cultural differences in your audience can seriously affect the usability of a product.

“A subtle issue that seems to figure in many accidents is social pressure. Although at first it may not seem relevant to design, it has strong pressures on everyday behaviour”

The book then moves on to suggest a system for design, and how to think about important aspects of your product like constraints, affordances and signifiers.

In the chapter on design principles for dealing with error, Norman makes an interesting point that I think is often at the centre of problems with computer systems:

“Difficulties arise when we do not think of people and machines as collaborative systems, but assign whatever tasks can be automated to the machines and leave the rest to people”

This statement reminded me of a paper I’d read a couple of years back, written in the 1980s, about automation: “The designer’s view of the human operator may be that the operator is unreliable and inefficient, so should be eliminated from the system. There are two ironies of this attitude. One is that designer errors can be a major source of operating problems”. As creators of computer software designed to automate tasks for us mere humans, we’d do well to remember that statement, and Norman’s view that, accurately, we are invariably collaborating with a computer program.

In a reflection of what is often at fault with software design, is the section titled “Solving the correct problem”.

“Good designers never start by trying to solve the problem given to them: they start by trying to understand what the real issues are.”

A fundamental part of that is asking the right questions. In many cases, I’ve seen user case study questionnaires built around questions that simply prompt answers the developer/company wants to hear. The company designed the questions about their own thinking. It’s hard to come up with questions that are rooted in trying to understand things you may not know about! But, there are skilled UX researchers out there who can.

One last thing that may not come as a surprise, especially if you’ve read Kahneman or Sutherland. The opening to the acknowledgements section:

“The original edition of this book was entitled The Psychology of Everyday Things (POET). This title is a good example of the difference between academics and industry. POET was a clever, cute title, much loved by my academic friends. … the editors also said, “But of course, the title will have to be changed.” Title changed? I was horrified. But I decided to follow my own advice and do some research on readers. I discovered that while the academic community liked the title and its cleverness, the business community did not. In fact, business often ignored the book because the title sent the wrong message.”

After reading DOET, I would add it to my essential reads list for building better products, teams and companies.

Thanks for reading this post via RSS! 👏

I have a feeling that’s been a question on all our minds since the early to mid-1990s. The Internet has given us so much utility for free. It’s amazing how blind we’ve been to ‘free’ ultimately having a cost though. Google, Facebook, et al. — all that freeness has come at a cost.

For some time I’ve looked for alternative ways to achieve things on the Internet without having to freely give up data to every advertising network. Last summer I tried out Hey for email and stuck with it as a paying customer. I can imagine many people asking themselves “why would you pay for an email service?!” Well, I find a lot of value in their raison d’être and I’m happy paying the cost for that value.

Hey made me realise how email can be a better experience when it’s done simply and thoughtfully, so I wondered where else I could do better with my digital life. I’m developing a project at the moment that has a web presence, but I don’t want to soil it with cookie notices. They’re a massive annoyance that only exists because web sites are busy farming our data. But, as a business, you really need some data about your customers! The key thing I wanted was visitor data. For years I’ve used Google Analytics, but Google is a prime example of how uncomfortable I’ve become with ‘free’. So I looked around for alternatives.

Many years ago I remember using AWStats for gathering web visit statistics, but it’s clumsy and looks dated now. I wanted something as simple and elegant as Hey. After some searching, I stumbled across Plausible Analytics.

Plausible has a simple and tidy interface. First goal met.

The weekly summary email is a useful addition

As I looked into the company, I discovered it’s not just the product that’s impressive. Two guys, bootstrapped — no VC money to poison the way they achieve growth. They’ve done ‘honest marketing’ — helpful blog content coupled up with a product that transparently ‘does what it says on the tin’

I love this blog post they wrote in 2020 — 15 best startup marketing practices we say no to (while growing our MRR by 1000% in 6 months).

The icing-on-the-cake for me to become a paying customer was that article.

Sometimes, value outweighs cost. A small monthly cost brings the value of a better experience for viewers of this site, and the data is not blindly being handed to advertisers.

Thanks for reading this post via RSS! 👏

In a word, yes. But the question needs some context.

Let’s start with defining “mindfulness”. Wikipedia, the UK charity Mind, and the UK national health service all say a similar thing, that “mindfulness” is awareness, a sense of what’s happening in the moment. Meditation is the tool we can use to bring about mindfulness. As Colm Meaney’s character in Layer Cake, Gene, succinctly puts it — “meditation is concentrating the front of the mind with a mundane task so the rest of the mind can find peace”. Usually that focus is on our breathing.

The first thing you notice when you meditate is just how busy your mind is. It’s like trying to get a dog to sit still so you can take its photo whilst there are squirrels running around. This is the first, and most valuable, thing you learn from the process — being aware of your mind. Robert Wright:

So when my turn came to speak, I gave voice to this frustration. The ensuing dialogue with my teacher went something like this:
So you notice that your mind keeps wandering?
Yes.
That’s good.
It’s good that my mind keeps wandering?
No. It’s good that you notice that your mind keeps wandering.
But it happens, like, all the time.
That’s even better. It means you’re noticing a lot.

Wright talks about the default mode network, an area of our brains best known for being active when we’re not focused on the outside world and the brain is at wakeful rest. Being able to even notice this wandering is happening and to bring it under control is just the start of what mindfulness brings.

A few years ago my wife did an online course on mindfulness, and at the time I pooh-poohed it. Thinking of myself as being of a more scientific mind, something as woolly as meditation didn’t fit. But then I suffered what I came to realise was burnout. I decided to be a little more open-minded and downloaded the Headspace app on my phone.

I’ve always found it hard to stay focussed on anything; it’s probably how I ended up in product management 😃 Juggling lots of things at once suits the way my mind wanders (side note: writing blog posts is hard work). I am well aware I do not have a restful mind.

I digress. I started using the Headspace app with a minimal 10 minute meditation. Determined to ‘play along’ with the gamification, I wanted to keep a streak going for as long as possible. I managed a year. 10 minutes, every single day, for 365 days. For starters, it was purgatory — even 10 minutes felt like forever, and nothing ever seemed to happen. Then one day I realised I was noticing things. I was noticing my mind wandering off, noticing feelings bubbling up — like frustration or anger. Although hard to describe, it’s like suddenly having two minds — with one monitoring the other if it’s running away.

“If you just sit and observe, you will see how restless your mind is. If you try to calm it, it only makes it worse, but over time it does calm, and when it does, there’s room to hear more subtle things–that’s when your intuition starts to blossom and you start to see things more clearly and be in the present more. Your mind just slows down, and you see a tremendous expanse in the moment. You see so much more than you could see before”¹

I recall the first time I had an experience directly related to having practiced mindfulness for a long period. I’d flown to New York for a 45 minute presentation to a client, part of an important deal a colleague had been working on. That morning I did my usual 10 minute Headspace before I left the hotel. During my presentation, something very odd happened. I could almost “see” thoughts coming, as if they were on a road ahead of me and I was following them. The words I knew I wanted were appearing in my mind as if on a teleprompter — it’s quite hard to describe, and I’d think nothing of it if it wasn’t for the fact I’d experienced nothing like it before. A couple of years later a friend recommended I read Robert Wright’s Why Buddhism Is True (she knows I’m not at all spiritual, so knew this book would be good — she was right. I highly recommend reading it) and as I read this bit, I realised what I’d experienced:

“Come to think of it, I had never seen this thought assume any form at all. But it now looked as if—literally looked as if—one part of my mind was speaking the thought to another part. There was even a kind of line tracing the path of the message, like an arrow on a diagram indicating the direction of communication. I watched this intracranial conversation, watched the message travel from sender to recipient, as a kind of outside observer, even though I thought of the recipient as in some sense being me. It’s almost impossible to convey in words the power of this experience and its aura of significance.”

He later talks about how chasing this kind of experience will only leave you disappointed. This is something that I’ve learnt from meditation, it’s best just left to do its thing. Have no expectations. It’s a system, not a goal. Just do it and see what happens.

Wright’s book is full of science, and Headspace have their own information on the details we now know about meditation’s physical effects on the brain. Recently I came across this article, about Steve Jobs.

“There was one thing, however, that Jobs didn’t know, although he might have guessed it to be true. Meditation does more than just calm you down and make you a better manager. Meditation literally causes your brain to age backwards.”

Last year I probably meditated two or three times a week, but I was up to 20-minute sessions. I no longer need the guidance of Headspace, but I’ve stuck with it as there are some interesting courses. I’ve done the ‘motivation’ course four times already, and it’s one I return to again and again. Plus, Andy Puddicombe often imparts some little gems of wisdom in his relaxing tone of voice. I can’t meditate for less than 20 minutes at a time now, it feels rushed. Most days are pretty terrible, with my mind wandering all over the place. But now and then I have a fantastic session, and the entire day seems to unfold in a more pleasant fashion. I’ve started 2021 with the goal of meditating every day, for the entire year, once again.

I now find myself able to control my emotions, able to concentrate, able to see things far clearer than I could before. It feels easier to see a bigger picture — when I look at problems at work, I find my mind automatically ‘zooms out’ to find the wider context. My wife says I am more patient than I’ve ever been.

Mindfulness very much works for me. Will it work for you? To be candid, I can’t see a reason not to try. It’s a tool I would not want to be without.

Thanks for reading this post via RSS! 👏

I added a new camera to our home (Logitech Circle View — pleasant experience so far) but it wasn’t showing up on my wife’s iPhone. So I ended up removing her from the home access and tried to re-invite. The invite sent OK, the notification came up on her phone OK, but clicking accept did nothing. In the Home app itself the invite ‘accept’ button was missing, with a spinning icon there instead. No matter how long I left it, or whichever of the addresses associated with her iCloud account I tried, they all failed. I tried the iPad and the iPhone, to no avail. I tried logging out of iCloud on all her devices and back in again, which also failed.

This was beginning to get immensely frustrating, especially when searching for an answer I read on an Apple forum that you needed to delete the home and set it all back up again 😱 I was loathed to do that, as I have many devices associated. It would take hours.

Then I discovered an old comment on a Reddit thread, which said try signing out of iCloud on the invitee’s device, signing in with your own — known working — ID on their device, checking Home looks right, signing back out and back in as them and resending an invite.

Bingo! Finally, it works.

What a faff. Totally inexplicable too, which is the doubly infuriating part.

Thanks for reading this post via RSS! 👏

postfix/bounce[26281]: fatal: open lock file pid/unix.bounce: cannot open file: Permission denied

Even with my [extremely] rusty tech skills I figured I could fix this one. postfix set-permissions is simple enough. Only, this didn’t fix it. A quick search showed a few promising results — it’s an SELinux problem. Almost every result therefore said setenforce 0 solved it!

That’s not the right answer. SELinux is very useful to have in place for anything connected to the internet, so rather than disable it, I looked into fixing the issue properly. Luckily I’ve been here before, many years ago. So referring to my previous blog post about SELinux, it took me all of 10 minutes to fix.

sesearch -s postfix_bounce_t -t var_run_t -c dir -A
ausearch -m avc --start today | audit2allow -m postfix_pid_write -o postfix_pid_write.te
ausearch -m avc --start today | audit2allow -M postfix_pid_write
semodule -i postfix_pid_write.pp
semodule -l | grep postfix

Problem solved. SELinux still very much in place.

Thanks for reading this post via RSS! 👏

From 2014 until early 2020 I was working for a US company, where initially I was the only employee in Europe. Inevitably communication was either asynchronous — via Slack — or by video meeting.

2020 has seen a step change in the way we communicate, and we’ve learned a lot about people’s shelving. But mostly, I’m still seeing a lot of scope for improving the way we use video meetings.

I think a lot of the fatigue we’ve heard people complain of is down to the lack of direct human connection, and it’s exacerbated by the way we use our computers.

So I recorded a small video of some tips I’ve found to improve the video meeting experience. A lot of the tips don’t require spending any money either…

Thanks for reading this post via RSS! 👏

“Good storytelling never gives you four, it gives you two plus two… Don’t give the audience the answer, give them the pieces and let them conclude the answer”

Have you ever read a Michael Lewis book? Liars Poker, The Big Short, Flash Boys? They’re all books about the finance industry. I’m not sure I can think of a duller subject to write about. However, if you have read any of his books, you’ll know they are riveting reads. Clichéd it may be, but they’re unputdownable. This is because Lewis is a great storyteller.

Throughout this series of reviews we’ve visited books on psychology, company culture and marketing. All these topics have the potential to be dry fact-filled subjects. But like Michael Lewis, each author is adept at telling stories, so the books are a compelling read.

In Into the Woods, John Yorke uses his 30 years’ experience of TV drama production to tell the story of the shape of all successful narratives. Without the ability to tell compelling stories, we’ll not be able to impart any of the information we’ve learned through the first four books.

The opening quote is from Andrew Stanton of Pixar, albeit paraphrased. It continues:

“Audiences have an unconscious desire to work for their entertainment. They are rewarded with a sense of thrill and delight when they find the answers themselves.”

What’s so interesting to me about this, is it ties in with things learned from Rory Sutherland “people seem to like choice for its own sake” and Daniel Kahneman “The more information, logic and reasoning you present to try to get an idea across, the more you actually push the audience away”. This is probably because you’re leaving the audience nothing to do, they don’t have to work for the answers.

The book goes into the basic structure of any story, as a commonly known three act structure. There is the thesis, the antithesis and the synthesis. Put another way in Hollywood simplicity terms:

Act One: Establish flawed character
Act Two: Confront them with their opposite
Act Three: Synthesise the two to establish balance

“The endless recurrence of the same underlying pattern suggests psychological, if not biological and physical reasons for the way we tell stories. If we don’t choose to tell them that way, perhaps we are compelled to.”

This statement from the book made me think of Kahneman’s studies and Sutherland’s observations once again. Is there something built in to us that means a story structured in this way compels us more? If we break from that structure, do we make our story less effective?

“‘Story as such’, said E. M. Forster, ‘can only have one merit: that of making the audience want to know what happens next. And conversely it can only have one fault: that of making the audience not want to know what happens next.”

This is the famous Volkswagen Beetle advert. It’s widely recognised as a genius piece of marketing. “The advert sucks you in; it impels you to do the work…because it generates conflict and the desire for its resolution”. It is model storytelling on a single page.

Being cognisant of giving too much information, here are two final pieces of useful material. One, another book — Resonate by Nancy Duarte is great if you need to present at all. It’s a great guide to making compelling presentations. Speaking of which, here’s a brilliant startup pitch. Can you see the clear three act structure?

I’ve read these five books over the course of the last three or four years, starting with Creativity, Inc, then Into the Woods. The last of the five was Rory Sutherland’s Alchemy, which I read last year. I had no idea they would interweave in the manner they do, and none of them came to my attention through the forced serendipity of a digital algorithm. It is only with hindsight that I realise they lay such good foundations for achieving almost anything in the modern workplace. A simple set of rules, a manifesto if you like, could easily be formed using ideas in these five books from which to build products, teams or even whole companies. Maybe that’s a future post…

Thanks for reading this post via RSS! 👏

“Things are not what they are; they are what we think they are”

My wife is a cryptic crossword fan, and she’s very good at them. I’ll confess, I am hopeless at them. The secret to cryptic crosswords is not to take the clues literally. There are patterns and rules that can lead you to the answer, and there are always signatures or cues to the crossword setter. The reasons for an answer may not be all that obvious.

So if “reality” is only what we think it is, does that mean there is scope for changing perception without having to actually change anything? Can a little ‘cryptic crossword magic’ apply to the everyday? Rory Sutherland seems to think so, and his book ended up being the most enjoyable read I had last year.

Thinking differently

“To avoid stupid mistakes, learn to be slightly silly”

Throughout the book Sutherland gives examples of “psychological moonshots”. Take the Uber map — it doesn’t reduce the waiting time for a taxi but makes waiting less frustrating. His TED talk from Athens is worth a watch. Just view the next minute or two from here (the link should take you to 6:22 in) where he talks about how we could’ve saved billions on the Eurostar HS1 train link. OK, so it’s a little flippant, but it’s an alternative way to get us to look at the same situation differently.

As with the other three books visited up to now, Sutherland tells a great story. Throughout the book there are more examples like his HS1 suggestion that were less flippant and actually used (charity mail shots and pension planning, to name two seemingly dull topics brightened by a little alchemy). He’s also visited a lot of the same books as I’ve been drawn to in this series, often referencing Kahneman’s material, for example. 

One book he mentions I’ve not read — The enigma of reason by Dan Sperber and Hugo Mercier. I’m sure we’d all like to think we make decisions by deductive reasoning, but Sperber and Mercier came to a different conclusion:

“One astonishing possible explanation for the function of reason only emerged about ten years ago: the argumentative hypothesis suggests reason arose in the human brain not to inform our actions and beliefs, but to explain and defend them to others. In other words, it is an adaptation necessitated by our being a highly social species.”

This explains another of Sutherland’s comments, which I’m sure all of us in corporate life have sadly experienced at one time or another:

“what often matters most to those making a decision in business or government is not a successful outcome, but their ability to defend their decision, whatever the outcome may be.”

We are so often led down the garden path by the need to appear logical and scientific, when a better result may come from thinking quite the opposite. And this is what Sutherland attempts to get us to do throughout the book (“This book is intended as a provocation”). As he says very early in the book, “the opposite of a good idea can also be a good idea”.

It takes courage to think differently to the crowd, but if you don’t think differently, how will you stand out when you need to? New products, companies and even working methods in existing organisations will require different thinking. The ideas in this book, coupled up with Loonshots, should be preparation enough for most of us.

Let’s leave Alchemy with a subsection heading from the book. The importance of this can’t be understated:

“Context is everything”

Come back next time for the final part in the series, when we’ll look at John Yorke’s Into the Woods, a book about storytelling.

Thanks for reading this post via RSS! 👏

Loonshot: a widely dismissed idea whose champions are often written off as crazy.

If you’ve ever worked in a corporate, I’m sure you’ll have heard somebody at some point say: “that’s not the way we do things around here”. The culture of maintaining the status quo brings about fixed mindsets, yet if there’s one constant, it’s change. To keep up in today’s technology-led, ever changing world, a business needs to be innovative.

Loonshots addresses the disconnect between being innovative and trying to maintain stability. These are two different worlds, with different mindsets. Through a variety of stories and characters, Safi Bahcall shows how large organisations in the past have successfully created ‘loonshot nurseries’. 

You could easily use Loonshots as a manifesto for how to innovate inside a large organisation. The appendix of the book even lists a set of rules for guidance. But it’s not just about enabling innovation, the ideas within Loonshots can help with bridging any disparate worlds.

Without listing the rules (other blog reviews have, I’ll not detract from the value of reading the whole book though) there are a few valuable lessons in Loonshots.

There are large differences in culture between corporates and startups, dictated by a simple idea — Dunbar’s Number. Bahcall discusses Dunbar’s Number, and how some organisational structure changes can remove the barrier it creates. There’s a very interesting chapter on DARPA, for example, which shows ‘soft equity’ as a motivation tool.

Speaking of motivation, reading that chapter reminded me of Dan Pink’s the puzzle of motivation talk. It’s worth a watch if you’ve not seen it before.

“Separate your artists and soldiers”

With his background in science, Bahcall naturally references his knowledge throughout the book. He compares the shift away from an innovative company to a phase transition. Put another way, a tipping point. In trying to recapture the innovative spirit, he recommends forming exclusive teams, with the creative inventor types separated from the day-to-day operations.

That should immediately strike you as a recipe for disaster. It’s highly likely to create a ‘them and us’ culture. Which is where his next historic example comes in, from the mind of Vannevar Bush, ‘create dynamic equilibrium’. Create a separate team, department or subsidiary, but make sure it, and the parent, are in constant communication. This reminded me of a TED talk I’d watched some years ago by Yves Morieux, titled “As work gets more complex, 6 rules to simplify” (if you’d like to read rather than watch, there’s a blog post by Yves too). Rule number 1: understand what your colleagues do.

“And then we’ll see how small changes in structure can transform the behaviour of groups”

If you work in the technology industry, you’ll see a parallel that was a great idea but has failed to deliver for many; DevOps. The original ethos was about connecting the stability seeking ‘ops’ world with the innovation-chasing ‘dev’ world. Products and consultancies sprang up, promising to bring you the all important innovation through DevOps. Yet most miss the key part of the puzzle, people. DevOps teams become a new silo, with the dynamic equilibrium sadly absent.

Bahcall and Morieux have one thing in common: their ideas learn from Vannevar Bush’s thinking. It’s funny that we’d solved a lot of these problems 80 years ago, isn’t it?

In the next post of the series we’ll look at Rory Sutherland’s Alchemy. This was the most enjoyable book I read in 2019. It made me realise something so obvious—every coin has two sides. If you can’t change the situation, reframe it.

Thanks for reading this post via RSS! 👏

“Figuring out how to build a sustainable culture … wasn’t a singular assignment. It was a day-in-day-out, full-time job.”

Ed Catmull is the retired former president and co-founder of digital animation studio, Pixar. Creativity, Inc. is the story of how Pixar overcame the challenges all businesses face to create and maintain a successful culture as the company grew. A culture that could outlast its founders.

To say there has been a lot written about company cultures is an understatement. 4.2 billion hits in a Google search!

I’ve had a bee in my bonnet about company culture for a while. Having started several times to write a blog about it, I couldn’t nail what I wanted to say succinctly enough. The opening quote on this post succeeds where I struggled. That’s the nub of the issue—you can’t just set your company culture up with some words and expect it to survive as you grow. Company culture is like a relationship; it needs ongoing attention to keep it alive and well.

Reviewing my notes for Creativity, Inc. I see there are 96 highlights! This is by far the most highlights I’ve made in a book in recent times. So much of Creativity, Inc. struck a chord with me.

“Even now, forty years later, I’ve never stopped questioning.”

This statement on its own should be a guiding light. Never stop questioning. At some point in our adult lives that seems to happen though. People accept the status quo and chug along. But building a company can’t be like that. Any startup needs to be questioning — with good intent — everything that has gone before, else they cannot hope to compete. How do you differentiate yourself from companies in the same space if you do as they do?

Mr Catmull, again:

“The pricing advice I was given—by people who were smart and experienced—was not merely wrong, it kept us from asking the right questions.”

This complements a section of Thinking, Fast and Slow, where Mr Kahneman talks about ‘anchoring’. It’s important to recognise this can happen and develop a technique for keeping one’s own bias in check.

With that in mind, one of the greatest lessons I took from Creativity, Inc. was about drawing. As part of learning how to use their software internally, they also hired an art teacher to teach people how to draw. There was more to this than first meets the eye:

“Another trick is to ask students to focus on negative spaces—the areas around an object. For instance, in drawing a chair, the new artist might draw it poorly, because she knows what a chair is supposed to look like (and that chair in her head—her mental model—keeps her from reproducing what she sees in front of her). However, if she is asked to draw what is not the chair—the spaces between the chair legs, for example—then the proportions are easier to get right, and the chair itself will look more realistic.”

The idea behind this exercise is to help the brain see what’s really there, and not what it thinks is there. It’s easy to fool the brain, bcse it wnts to mk snse f th wrld, evn f it dsnt mke snse. I love the metaphor of drawing around the chair and frequently use it to challenge my thinking.

Building a company culture is one thing, maintaining it is a whole other mission. Creativity, Inc. goes into how Pixar recaptured their original culture, after the founders realised it was slipping away as the company grew larger. There is a great deal of sound advice in this book, and as expected of master storytellers Pixar, it’s told well. I’ll leave you with one last quote:

“To me, the answer should be obvious: Ideas come from people. Therefore, people are more important than ideas.”

In part three we’ll review Loonshots by Safi Bahcall.

Thanks for reading this post via RSS! 👏

“The world makes much less sense than you think. The coherence comes mostly from the way your mind works.”

Following on from the introduction post, if there’s one important takeaway I learned from reading Thinking, Fast and Slow by Nobel Prize winning Daniel Kahneman, it’s don’t always trust whatever thought first comes to mind.

While reading this book I enjoyed the insight into how my own thinking works. Kahneman’s approach of telling a story about our brains as if they’re two characters, which he calls System 1 and System 2, goes a long way to showing how our subconscious mind can easily fool our decision making.

I’ve long been fond of challenging my initial gut feeling on anything, but this book introduced me to the premortem. Attributed to psychologist Gary Klein, the premortem could be a saving step for any project:

The premise of the session is a short speech: “imagine that we are a year into the future. We implemented the plan as it now exists. The outcome was a disaster. Please take 5 to 10 minutes to write a brief history of that disaster.”

This forces us to think of the exact opposite of what we’ve been planning for. An imaginative way to keep our assumptions in check.

One powerful example of our natural tendency to think irrationally is the Linda problem, otherwise known as the conjunction fallacy. Before you head off to the Wikipedia page, test yourself:

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which is more probable?

1. Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement.

Which did you pick? Now you can read the Wikipedia page.

Improving your influencing skills

You’ll find a lot of blog posts written off the back of Thinking, Fast and Slow. Tyler Odean, a product manager at Reddit (and formerly, Google), uses Kahneman’s writing as the base for an article on improving your influencing skills.

“The more information, logic and reasoning you present to try to get an idea across, the more you actually push the audience away”

I found Kahneman’s book to be a valuable read, not only because it’s given me a new insight into my own thinking. It has also given me the ability to see everything has a reason—even when one doesn’t appear clearly or immediately. The book brings so much more weight to a statement I’ve sworn by for years:

K I S S: Keep It Simple, Stupid

Next time we’ll take a look at Creativity, Inc by Ed Catmull.

Thanks for reading this post via RSS! 👏

“The human mind does not run on logic any more than a horse runs on petrol.”

This quote, from Rory Sutherland, has become like a pebble dropped into a pond for me. Having spent most of my career working in the technology industry, logic has led me everywhere. However, I worry logic and data have often blinded me to simple truths. We can solve every problem with logic. Except now I’ve read these five books, I realise logic has little to do with it.

Interwoven across these five books is a recipe for building successful companies, teams and products. They’ve given me a deep understanding of a journey I experienced, but didn’t fully comprehend at the time.

Having worked in some of the largest corporations in the world and in startups, I now realise they are different worlds. They feel almost exclusive, where each world will advertise jobs requiring experience of their world. Today, more than ever, that’s a mistake. Because each world can learn from the other, and to keep up with the rapid pace of technological progress, they should.

For the sake of brevity, today I’ll just introduce the books. In a set of follow-up posts, I’ll go into more detail about their relevance and the points I found most helpful.

1. Daniel Kahneman - Thinking, Fast and Slow
2. Ed Catmull — Creativity, Inc 
3. Safi Bahcall — Loonshots
4. Rory Sutherland — Alchemy
5. John Yorke — Into the woods

I could probably extend this list out to 10 or 20 without trying too hard. I often refer to anecdotes from Matthew Syed’s Bounce and Malcolm Gladwell’s Outliers. In more recent years, I’ve also read David McRaney’s How Minds Change and found it to be immensely valuable, along with Don Norman’s The Design of Everyday Things.

These books have given me insight into why we all think the way we do, why I prefer working in forward-looking, agile, companies, how large companies can learn a lot from small ones (and vice versa) and how the immediate answer that comes to mind is rarely the one to take you forward in anything.

I’ve come away from reading each one of these books with renewed energy and a mind spinning with ideas. If you’ve read any of them too, I’d love to hear your thoughts on LinkedIn or Mastodon.

Thanks for reading this post via RSS! 👏

But for those of us working with estates larger than a handful of machines it’s not that simple. Sometimes an update can create unintended consequences across many machines and you’re left wondering how to put things back the way they were. Or you might think “should I have applied the critical patch on its own and saved myself a lot of pain?”

Faced with these sorts of challenges in the past led me to build a way to ‘cherry-pick’ the updates needed and automate their application.

A flexible idea

Here’s an overview of the process which we’ll apply technology to in a moment…

Under this system, we do not permit machines direct access to vendor patches. Instead, they’re selectively subscribed to repositories. Repositories contain only the patches required – although I’d encourage you to give this careful consideration so you don’t end up with a proliferation (another management overhead you’ll not thank yourself for making).

Now patching a machine comes down to 1) the repositories it’s subscribed to and 2) the ’thumbs up’ to patch. With variables controlling both subscription and permission to patch we need not tamper with the logic (the plays), we only need to alter data.

Here is an example Ansible role to fulfil both requirements. It manages repository subscriptions and has a simple variable to control running the patch command:

---
# tasks file for patching

- name: Include OS version specific differences
  include_vars: "{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml"

- name: Ensure Yum repositories are configured
  template:
    src: template.repo.j2
    dest: "/etc/yum.repos.d/{{ item.label }}.repo"
    owner: root
    group: root
    mode: 0644
  when: patching_repos is defined
  loop: "{{ patching_repos }}"
  notify: patching-clean-metadata

- meta: flush_handlers

- name: Ensure OS shipped yum repo configs are absent
  file:
    path: "/etc/yum.repos.d/{{ patching_default_repo_def }}"
    state: absent

# add flexibility of repos here
- name: Patch this host
  shell: 'yum update -y'
  args:
    warn: false
  when: patchme|bool
  register: result
  changed_when: "'No packages marked for update' not in result.stdout"

Scenarios

In our fictitious, large, globally dispersed environment (of four hosts) we have…

• Two web servers
• Two database servers
• An application, comprising one of each server type

OK, so the number of machines I’m showing here aren’t ’enterprise scale’, but remove the counts and imagine the environment as multiple tiered applications, dispersed geographically. What we want to do is patch elements of the stack; across server types, application stacks, geographies, or the whole estate.

Using only changes to variables, can we achieve that flexibility? Sort of. Ansible’s default behaviour for hashes is to overwrite. In our example the patching_repos variable for our db1 and web1 hosts gets overwritten by their later occurrence in our inventory. Hmm, a bit of a pickle. There are two ways to cater for this:

1. Multiple inventory files
2. Change the variable behaviour

I chose number one, because it maintains clarity. Once you start merging variables it’s hard to find where a hash appears and how it’s put together. Forcing yourself to use the default behaviour maintains clarity and is the method I’d encourage you to stick with, for your own sanity.

Get on with it then

Let’s run the play, focussing only on the database servers.

Did you notice the final step, ‘Patch this host’, says ‘skipping’? That’s because we didn’t set the controlling variable to actually do the patching. What we have done is set up the repository subscriptions, ready.

So let’s run the play again, limiting to the web servers, and tell it to actually do the patching. I’ll run with verbose output set, just so you can see the yum updates happening:

Patching an application stack has required another inventory file, as discussed above. Let’s run the play again:

Patching hosts in the European geography is the same ethos as the application stack, and we achieve it with another inventory file:

Finally, now all the repository subscriptions are configured, let’s just patch the whole estate. Note the app1 and emea groups don’t need the inventory here – we were only using the inventories for separation of repository definition and set up. They’re now configured, so the yum update -y patches everything. If you didn’t want to capture those repositories they could be configured as enabled=0:

Conclusion

The flexibility comes from how we group our hosts. But with the default hash behaviour we need to think about overlaps – the easiest way, to my mind at least, is with separate inventories.

With regards to repository set up, I’m sure you’ve already said to yourself, “ah, but the cherry picking isn’t that simple!” In this model there is additional overhead to download patches, test they work together, and bundle them with dependencies in a repository. With complementary tools you could automate the process, and in a large scale environment you’d have to.

There is a part of me drawn to just applying full patch sets as a simpler and easier way to go. Skip out the cherry picking part, and apply a full set of patches to a ‘standard build’. I’ve seen this approach applied to both UNIX and Windows estates in the past – with an enforced quarterly update.

I’d be interested in hearing your experiences of patching regimes, and the approach proposed here, via Mastodon.

Thanks for reading this post via RSS! 👏

Requirements

Bootstrapping hardware is mostly about network services. Before we can install an operating system (OS) then, we must first set up some services. We need:

• DHCP
• PXE
• TFTP
• OS Media
• Web server

Setup

Besides the DHCP configuration, everything else in this article is handled by the Ansible plays included in this repository.

DHCP server

I’m writing here on the assumption you can control your DHCP configuration. If you don’t have access to your DHCP server, then you’ll need to ask the owner to set two options. DHCP option 67 needs setting to pxelinux.0 and next-server (which is option 66 — but you may not need to know that, often a DHCP server will have a field/option for ‘next server’) needs setting to the IP address of your TFTP server.

If you can own the DHCP server, I’d suggest using dnsmasq. It’s small and simple. I will not cover configuring it here, but look at the man page and the --enable-tftp option.

TFTP

The next-server setting for our DHCP server, above, will point to a machine serving TFTP. Here I’ve used a CentOS Linux virtual machine, as it only takes one package (syslinux-tftpboot) and a service starting to have TFTP up and running. We’ll stick with the default path, /var/lib/tftpboot for this article.

PXE

If you’re not already familiar with PXE (preboot execution environment) you might like to take a quick look at the Wikipedia page. For this article I’ll keep it short–we will serve some files over TFTP, which DHCP guides our hardware to.

You’ll want images/pxeboot/{initrd.img,vmlinuz} from the OS distribution media for pxeboot. These need copying to /var/lib/tftpboot/pxeboot. The relreferenced Ansible plays do not do this step so you need to copy them over yourself.

We’ll also need to serve the OS installation files. There are two approaches to this. 1) install, via HTTP, from over the internet or 2) install, again via HTTP, from a local server. For my testing, since I’m on a private LAN, (and I guess you are too), the fastest installation method is the latter. The easiest way to prepare this is to mount the DVD image and rsync the images, Packages and repodata directories to your webserver location. The relreferenced Ansible plays will install httpd but won’t copy over these files, so don’t forget to do that after running the play. For this article we’ll once again stick with defaults for simplicity–so files need copying to Apache’s standard docroot, /var/www/html.

Directories

We should end up with directory structures like this:

PXE/TFTP

[root@c7 ~]# tree  /var/lib/tftpboot/pxe{b*,l*cfg}
/var/lib/tftpboot/pxeboot
└── 6
    ├── initrd.img
    └── vmlinuz

httpd

[root@c7 ~]# tree -d /var/www/html/
/var/www/html/
├── 6 -> centos/6
├── 7 -> centos/7
├── centos
│   ├── 6
│   │   └── os
│   │       └── x86_64
│   │           ├── images
│   │           │   └── pxeboot
│   │           ├── Packages
│   │           └── repodata
│   └── 7
│       └── os
│           └── x86_64
│               ├── images
│               │   └── pxeboot
│               ├── Packages
│               └── repodata
└── ks

You’ll notice my web setup appears a little less simple than the words above! I’ve pasted my actual structure to give you some ideas. The hardware I’m using is really old, and even getting CentOS 7 to work was horrible (if you’re interested; lack of cciss drivers for HP Smart Array controller — yes, there is an answer, but it takes a lot of faff to make it work) so all examples are of CentOS 6. I also wanted a flexible setup which could install many versions. Here I’ve done that using symlinks — this arrangement will work just fine for RHEL too, for example. The basic structure is here though – note the images, Packages and repodata directories.

These paths relate directly to the PXE menu file we’ll serve up, and the kickstart file too.

If you don’t have DHCP

If you can’t manage your own DHCP server, or the owners of your infrastructure can’t help, there is another option. In the past I’ve used iPXE to create a boot image which I’ve loaded as virtual media. A lot of out-of-band/lights-out-management interfaces on modern hardware support this functionality. You can make a custom embedded PXE menu in seconds with iPXE. I won’t cover that here, but if it turns out to be a problem for you, then drop me a line on Mastodon and I’ll look at doing a follow up blog post if enough people request it.

Installing hardware

We’ve got our structure in place now, and we can kickstart a server. Before we do though, we have to do some configuration of the TFTP set up to enable a given piece of hardware to pick up the PXE boot menu.

It’s here we come across a small chicken/egg problem. We need a host’s MAC address to create a link to the specific piece of hardware we want to kickstart. If the hardware is already running, and we can access it with Ansible, that’s great–we have a way of finding out the boot interface MAC address via the setup module (see the reinstall play). If it’s a new piece of tin, however, we need to get the MAC address and tell our setup what to do with it. This probably means some manual intervention–booting the server and looking at a screen, or maybe getting the MAC from a manifest or such like. Whichever way you get a hold of it, we can tell our play about it via the inventory.

Let’s put a custom variable into our simple INI format inventory file, but run a play to set up TFTP…

(pip)iMac:ansible-hw-bootstrap$ ansible-inventory --host hp.box
{
    "ilo_ip": "192.168.1.68",
    "ilo_password": "administrator"
}
(pip)iMac:ansible-hw-bootstrap$ ansible-playbook plays/install.yml

PLAY [kickstart] *******************************************************************************************************

TASK [Host inventory entry has a MAC address] **************************************************************************
failed: [ks.box] (item=hp.box) => {
    "assertion": "hostvars[item]['mac'] is defined",
    "changed": false,
    "evaluated_to": false,
    "item": "hp.box",
    "msg": "Assertion failed"
}

PLAY RECAP *************************************************************************************************************
ks.box                     : ok=0    changed=0    unreachable=0    failed=1

Uh oh, play failed. It contains a check that the host we’re about to install actually has a MAC address added. Let’s fix that and run the play again…

(pip)iMac:ansible-hw-bootstrap$ ansible-inventory --host hp.box
{
    "ilo_ip": "192.168.1.68",
    "ilo_password": "administrator",
    "mac": "00:AA:BB:CC:DD:EE"
}
(pip)iMac:ansible-hw-bootstrap$ ansible-playbook plays/install.yml

PLAY [kickstart] *******************************************************************************************************

TASK [Host inventory entry has a MAC address] **************************************************************************
ok: [ks.box] => (item=hp.box) => {
    "changed": false,
    "item": "hp.box",
    "msg": "All assertions passed"
}

TASK [Set PXE menu to install] *****************************************************************************************
ok: [ks.box] => (item=hp.box)

TASK [Reboot target host for PXE boot] *********************************************************************************
skipping: [ks.box] => (item=hp.box)

PLAY RECAP *************************************************************************************************************
ks.box                     : ok=2    changed=0    unreachable=0    failed=0

That worked! What did it do? Looking at the pxelinux.cfg directory under our TFTP root, we can see a symlink…

[root@c7 pxelinux.cfg]# pwd
/var/lib/tftpboot/pxelinux.cfg
[root@c7 pxelinux.cfg]# l
total 12
drwxr-xr-x. 2 root root   65 May 13 14:23 ./
drwxr-xr-x. 4 root root 4096 May  2 22:13 ../
-r--r--r--. 1 root root  515 May  2 12:22 00README
lrwxrwxrwx. 1 root root    7 May 13 14:12 01-00-aa-bb-cc-dd-ee -> install
-rw-r--r--. 1 root root  682 May  2 22:07 install

The install file is symlinked to a file named after our MAC address. This is the key, useful, piece. It will ensure our hardware with MAC address ‘00-aa-bb-cc-dd-ee’ is served a PXE menu when it boots from its network card.

So let’s boot our machine.

Usefully Ansible has some remote management modules. We’re working with an HP server here, so we can use the hpilo_boot module to save us having to interact directly with the LOM web interface.

Let’s run the reinstall play on a booted server…

The neat thing about the hpilo_boot module, you’ll have noticed, is it sets the boot medium to be the network. When the installation completes the server restarts and boots from its hard drive. The eagle eyed amongst you will have spotted the critical problem with this–what happens if the server boots to its network card again? It will pick up the PXE menu and promptly reinstall itself. I would suggest removing the symlink as a “belt and braces” step then. I will leave that as an exercise for you dear reader :-) Hint: I would make the new server do a ‘phone home’ on boot, to somewhere, which runs a clean up job. Since you wouldn’t need the console open, as I had here to demonstrate what’s going on in the background, a ‘phone home’ job would also give a nice indication of the process having completed. Ansible, naturally. Good luck!

If you’ve any thoughts or comments on this process, please let me know.

Thanks for reading this post via RSS! 👏

TL;DR It’s OK. I’m getting by, with some slight working practice changes. No current need to buy a laptop.

The longer story…

If you read back through this blog, or you already know me, you know my background is twenty years of being a tech engineer. That changed a little over 4 years ago. It’s this which led me to think I could do without a laptop, and lighten the load in my luggage when I travel (which I still do a lot).

My day-to-day comprises Slack, email, conference calls over various medium (mostly Bluejeans and FaceTime) and Google apps like docs, sheets and slides. All of these are catered for well on iOS, and since I’m mostly only using one at a time, having it fill the screen, unlike a traditional desktop, is “fine”.

I say fine like that because it’s probably the first change to my working practices I’ve had to make and one I’m unsure yet is a good or a bad thing. I’ve alternated between writing this on my iMac–where, as anybody working on a traditional desktop will probably recognise, the screen is a splatter-cast of open windows and applications–and on the iPad itself.

On the iMac I’ve got windows open left, right and centre. I don’t even know why–but because I can’t do that on the iPad, I don’t. It’s probably a good thing though–I focus on one thing at a time, which for my mind is no bad thing. But it is a big shift in the way I’ve worked for a long time, and I can see why that might irk people.

The flip side of focussing on the one thing like this is if I need to refer to another application then it’s more of a pain on the iPad. Yes, you can do the split window thing, but it doesn’t work for all apps. And as I wrote that I promptly split the screen between Safari, where I’m writing this in an online editor, and Notes, where I’ve been keeping a tally of the annoyances :) So even as I describe this as a pain, I find it’s actually OK. Ha!

Other things that work ‘OK’, with minor annoyances. I recently gave a sales presentation, which I’d written in Keynote (I do use the Google Suite, but I still find Keynote the simplest presentation application). The actual presentation went fine–I plugged the iPad into the meeting room HDMI (using this adapter) and had a ‘presenter screen’, complete with notes, on the iPad and the content on the big screens in the room. It worked just as well as using a laptop. However, it wasn’t so easy to put the presentation together on the iPad, so I did most of the work on the desktop. You can put a perfectly workable set of slides together using Keynote for iOS, but I found accurately laying out text boxes etc. hard (I couldn’t find the actual pixels box on Keynote for iOS, for example–and who wants the horror of a text box jumping a couple of pixels between slides?!)

2024 Update: found the granular sizing and position controls!

Saving an email as a PDF is doable, but a faff, and not obvious.

When I’m travelling, I tend to snapshot receipts for expenses into Evernote. Then, because our expense system is archaic, I have to separately submit them as PDFs. Which is easy to do with the desktop version of Evernote because you can simply right click and choose ‘convert note to PDF’ in the context menu. Try doing that on the iOS version! Basically, you have to go through the same process (linked above) as printing an email to a PDF.

Writing for the blog is doable. I use the static site generator, Hugo and store all the pages in a private Git repository. Mostly I author in ProWritingAid and then paste the text into Working Copy. This handles the Git bit and is the closest I get to anything still remotely technical. Working Copy is a decent source code management solution for iOS, so would also be good for coders. Blog deployment happens via a GitHub Action.

Occasional VPN access is handled by the iOS Passepartout VPN client. There are a few options for SSH clients, and over the years I’ve swapped, but am currently using Termius (the free version is good enough).

One joy I’ve found is photo-editing. Photography is a minor hobby, and sometimes the digital pictures out of my Fuji cameras need some tweaking. Using Affinity Photo with the Apple Pencil on the iPad is so much easier than using either a mouse, or a tablet, on the desktop version.

The only total let down I’ve found so far, which will require a laptop purchase at some point again, is video editing. I use Final Cut Pro, which as of today does not exist for iOS. iMovie just doesn’t cut it (pun not intended). After a recent ski trip I stuffed days’ worth of filming into the Mac to edit and now I can’t be bothered to do it. Having a laptop when travelling means edits, or at least picking out favourite scenes to use later, is so much more productive when done later the day they were filmed¹.

The final total annoyance, for me at least, is fingerprints on the screen. I‘ve always hated it when folks point at something on your screen and smudge their fingers all over it, so to spend my days doing the very same myself is an annoyance I need to get over.

For now then I will stick with the iPad as my mobile compute of choice. At some point or another I will probably buy a laptop for the niche activities, but I’m in no rush.

Thanks for reading this post via RSS! 👏

“I do have better things to do you know” I replied.

“Yes, yes, I know. But who else is going to do it?”

“Give it here then!”

In the beginning, there was a problem

That was a fairly common interaction for me as a young lad. I was fresh out of school and working my summer in the sales department of a local car dealership. My job was mostly admin related tasks, which up until that point hadn’t included doing all the sales guys’ typing. Our secretary had recently departed the company, and the sales guys all figured I could happily do the replacement typing jobs. The duty had fallen to me because a) I had the stereotypical 1980s glasses of a nerdy computer kid and b) they all knew I actually was a nerdy computer kid. So fair play to them for assuming I could type, I could.

The thing was I really did have better things to do, and these daily interruptions were eating into my productive time. I wanted that time back; so I got a book on programming VBA (everyone remember that? yes? jolly good) and started to write some macros in Word to generate invoices. After a short while I had a simple dialog box, that even the oldest curmudgeon in the office could work, which filled in a neat looking invoice and printed it out. The interruptions stopped, the sales folk could all help themselves to get the job done, and I got countless hours of my life back.

That was the start of my IT career – automating. Solving problems for people using technology.

The runbook

Over the course of the last twenty odd years I’ve worked in a lot of companies, and yet even today I still see places doing things manually. As recent as last year I saw a ‘runbook’ in action – basically a list of manual steps somebody had to do to deploy code into production. As I’m fond of saying, “if the solution is delivering a business goal then it’s not ‘wrong’ per se, but sometimes things can be done better”. Following a runbook leaves so much to chance, one can never really be sure what the result will be.

Image source: wikipedia.org/wiki/Decrepit_car

Automation is about trust. We can have faith in task consistency with automation. Things turn out more the way we expect them to, time after time.

Image source: porsche-leipzig.com

I’ve visited customers and seen plenty who still aren’t trusting the computer to do the heavy lifting. Pondering why this might be, I’ve drawn four conclusions:

1. Speed of entry – “oh it’s quicker if I just type this straight into the server now”
2. It’s too hard
3. I only need to do this once
4. But if I automate this, won’t my job go away?

Number two is a tooling problem – if it’s too hard, then the tool is wrong for the job (ever tried putting a screw into a wall with a hammer?) Number three always makes me smile, because who ever heard of a temporary solution in IT? We all know that “bits of sticking plaster” end up running production systems. So I always think, automate it from the start and it’s repeatable and documented.

Number one is actually fed by numbers two and three. Number four is a bit of a fallacy too – automation isn’t always about replacing people, it’s about augmenting them. Get some of your own time back as an engineer and you are free to find more ways to be productive (I’m sure we’ve all heard of Google’s 20% rule and the products that originated from that free thinking). Every business needs to grow, and if you find time to investigate new ways to enable that growth then all the better.

Augmenting DevOps

But how does automation feature in the DevOps story? Well, “DevOps” as a word is a contraction of two words - Development and Operations. It was coined, supposedly, around the late 00s by a developer looking to get his code into production a lot faster. The two departments have opposing goals - development want to get their shiny new code out in front of people as quickly as they write it, yet operations want a stable environment for customers to visit (and let’s face it, as an operations person, who wants that dreaded call at 3am when something goes terribly wrong?). The defining ethos of the word was, in effect, we should communicate. The proposal was that Dev and Ops should stop working as discrete silos, where one simply throws their work over the fence to the other, and should start talking to one another to build “pipelines” – the software equivalent of assembly lines.

Some might say, therefore, that DevOps is common sense. To an extent, it is. When we’re a team, and every business is, we should remember we’re all in the same boat and rowing together in the same direction is going to get us places a lot faster. But often that’s hard to do in a large organisation because of many factors – geography, legislation, process etc.

Have you heard of Dunbar’s Number? I’ll quote the underlying Wikipedia link there - “…is a suggested cognitive limit to the number of people with whom one can maintain stable social relationships. These are relationships in which an individual knows who each person is and how each person relates to every other person…with a commonly used value of 150”.

Getting things done as a team is so much easier when everybody knows each other that little bit better. This is why so many corporates force upon us the often cringey team building days :) But they have merit. Learn a little bit of the social side to your colleagues and sure enough, you’ll work together easier. The problem with this magic 150, and the ethos of DevOps, is quite clear to see when you work in a massive, global, corporate though. How do you achieve the sort of relationships a close knit community enjoys? The DevOps ethos was formed out of people working in small, often startup-sized, companies.

I watched an interesting TED talk a while ago by a chap called Yves Morieux, titled “As work gets more complex, 6 rules to simplify”. I highly recommend watching it. His first rule is this:

Understand what your colleagues actually do

Coming back to the core tenet of DevOps, communication, one can see how this rule plays a key part. Once developers and operations understand each other’s working lives, it makes it easier to have the conversation about implementing a trusted, automated, pipeline. Sidenote: on the subject of working together, I read an excellent book many years ago called “Getting to yes” by Roger Fisher and William Ury. It talks about objective criteria rather than digging into positions. It’s well worth a read.

Looking at this in the context of a globally dispersed corporate organisation, it is possible to encourage, or bolster, the ability for colleagues to understand one another’s jobs using technology.

One of my favourite quotes is from Einstein: “make things as simple as possible, but not simpler”. Technology, over the last 20 odd years that I’ve been working with it, hasn’t always got simpler. On the contrary, a lot of the time I find things have got much more complicated, and often harder with that. Using technology to help understand what your colleagues do needs to be easy, because then it becomes far more natural to do so. The world now moves at a rapid pace of change and attention spans are short (thanks Twitter!), so information needs to be quick to access, easy to keep up to date, and most importantly easy to search for relevance. Wikis are a great documentation tool, just look at the success of Wikipedia.

Later in his talk, Yves Morieux goes on to speak about feedback loops. Reinforcing his first rule is tremendously beneficial to this communications story, and I’ve seen companies do this very well with technology. I once visited a customer who showed me their excellent Ansible development workflow. Anybody could write playbooks and contribute to an internal repository via a “pull request”. Essentially they had mimicked, internally, a typical opensource software development model you would see on GitHub. There was a feedback loop – people wrote code, and submitted it for others to review and bring into the bigger picture. Dispersed teams, geographically in this particular case, working together.

I’ve seen similar arrangements taken a step further, with commit messages and PRs automatically sending a message into a chat system. Within the Ansible team we’re big users of Slack. For a time when we were just a small startup it was my personal saviour. Although I was over 3,000 miles away from HQ I always felt very much connected to the organisation. Slack was magical at helping me understand what my colleagues did (in true BBC style – there are other chat systems available. I say this with some jest; as a veteran Internet user I’ve been using IRC for over 20 years, plus countless other chat systems along the way. Slack’s differentiator is they’ve made using it easy and relatively simple).

Chat systems can take feedback loops and automation to the next level too, with ChatBots. I’m sure we’ve all read about bots and how they’re revolutionising our interactions with companies, but it’s always impressive to see a company implement a ChatBot that can be asked to spin up 50 new web servers on demand.

The journey from legacy to DevOps

The migration of old methods to the modern DevOps mindset can be taken in small steps by introducing simple, easy-to-use technology that puts people to the fore. When solving a problem using a piece of technology that’s quick and easy to get started with, it’s far more likely to achieve widespread adoption. See how easy it is to edit a wiki page, put some code into a GitHub repository, create a Slack team or, and it would be remiss of me not to say, write an Ansible playbook, and you’ll witness how small seeds grow into big trees. When technology isn’t restricted to one particular group then it’s easy to introduce to many teams at the same time. It helps them work to their own cadence in parallel and also to come together in one, coherent, whole.

I’ll leave you with one last anecdote from my past. Many years ago I fought hard to get a wiki introduced into a large corporate I was working in. The documentation practice at the time was weighty, and often unread, Word documents. These documents quickly went out of date, and people couldn’t easily find information contained within them from a central location. After the wiki was implemented, more documentation was completed in the following six months than had been written in the past three years.

Rapid progress brought about by ease of use and simplicity.

Footnote: I originally wrote this blog in my day job for Ansible by Red Hat. It featured on ansible.com/blog. If you saw my talk at IPExpo London in 2015 you will also recognise it.

Thanks for reading this post via RSS! 👏

Mostly I talked about the core idea of DevOps being communication, about breaking down silos. I went on to suggest ways technology can support that idea.

Postscript: I received this email from the event organisers a month after the event.

The TED Talk mentioned in the slides: Yves Morieux, as work gets more complex, 6 rules to simplify.

Before we get going — who works in an organisation of less than 150 people? More than 1000? (Later on we'll come to Dunbar's number). #

I'm still seeing organisations using 'runbooks'; a list of things to do to deploy applications/systems. I've spent my career automating — sometimes upgrading manual processes to automated, and sometimes being lucky enough to do greenfield engineering. Repetitive, manual, tasks can be a massive drain on an organisation and detrimental to keeping the best engineers. #

Typing from a list of instructions _can_ lead to mistakes. We've all got stories of that one deploy 'fat fingered' and suddenly a whole production system gets killed. #

Automation ensures consistency. It shouldn't be a mystery why most auto manufacturing is robotised. #

Suggesting automation often gets the response "I'll just type it for now, it's quicker". They could've just typed the same into a Play. So it's caused to me to ponder why people would actively choose to make life harder for themselves. The obviousness of that being, they don't think they are making it harder. Quite the opposite. There are usually one of four reasons. #

The fourth one being the most powerful "won't I just automate myself out of a job?" (answer: no, but that's a bigger story) #

This was a graph a customer once showed me. "The green line is the business's expectation for revenue growth". "The blue line is the number of people we can have to do that growth". So how do we fill that gap? Automation. . . #

Ask the audience — who's heard of Dunbar's number? (At the time, this was delivered in front of an audience of roughly 200 people. Two raised their hands) DevOps — story of its origins. Dev & Ops pulling in opposite directions. Silos. Let's get them to talk! This should be common sense. #

I watched this TED talk with Yves Morieux a while ago, and love his first principle. #

Create digital 'feedback loops', so departments can work together, and learn from each other. A cohesive organisation, versus many small 'tribes' doing their own thing. A great collaborative example — Wikis versus old-fashioned Word docs. Arguably any online collaborative documentation tool is better! #

Another of Yves Morieux points was to create feedback loops — creating internal dev practices based on pull requests is a great example of this. #

As much as chat systems can be an overload of 'noise' (whole other conversation on the culture of implementing signal vs noise practices) they can be a good centralisation point for communications. #

ChatOps — use everyday communications tool with centralised automation #

Thinking back to slide 6, and the reasons why people don't automate — or adopt anything new for that matter — it's about making 'change' easy. So even when people default to resisting change, if you can demonstrate something productive within 10 minutes of picking up a product/technology, the resistance is far more likely to subside quickly. So "K.I.S.S.". With tech, the talented folks will prefer text based systems. Typing scales far better than moving a mouse and clicking. You won't build a creative organisation — and engineering is totally creative — with GUIs. Get stuff done in text, report on it in GUIs. #

This being the output of the previous slide 😂 Anyone remember Perl Golf? #

Common tool - everybody goes off to do their thing, with a clear direction. 'Meet in the middle'. #

Closing anecdote: IB Wiki (in 6 months more docs done than in previous 3 years - ease of use/access/finding information) i.e. ** make it easy ** // Create a collaborative culture by default - INS 'Knowledge Quests' #

Thanks for reading this post via RSS! 👏

Infrastructure as code has brought with it many good things from software development - not least the ability to define our infrastructures and ‘replay’ them, giving us the foundations of sound “Dev Ops” automation pipelines and the ‘cattle vs pets’ mantra. However, I don’t believe we need to take every last piece of development practices and apply to IaC.

When we model an infrastructure in something like Ansible we are, in effect, laying the foundations for our ‘house’. Get a bit of the foundations wrong, and the house won’t stand up. Less of the metaphor, getting literal - if your web server is configured wrong, it won’t serve web pages. But what if it serves a webpage, but not the webpage I hear you ask. Well, this is where the testing comes in.

A few years back I wrote some Perl code for a monitoring system. The code connected to a number of different kinds of VPN device, of varying versions, and scraped their web interfaces. The system I was replacing didn’t cope very well when versions of the VPN OS changed - it just fell over. I had written a lot of tests before the main coding, and a few of the tests checked for correct scenarios, and false scenarios. In effect I was looking at both sides of the coin - my thinking was in the future if the OS changed then the tests would reveal something that did or did not work (they were also useful to prove that version upgrades of the Perl module would still work correctly too). What I actually checked for was known good strings the monitoring software needed from the VPN devices.

This is my thinking around infrastructure code testing. Think big picture - test the goal you’re expecting to arrive at. Ansible modules are designed to ‘fail fast’ and achieve ‘desired state’. If they’re successful then you’ve implemented the state you want (put a file into place, template a configuration, ensure a service is running, etc). If the module doesn’t arrive at the desired state, it fails. If it fails, the run isn’t successful, and the goal you need to test for isn’t reached.

The point has also been raised about edge cases, such as ‘what if ignore_errors was used by accident, a copy n paste mistake, perhaps’. Well, again, I suggest test for the end goal. And I don’t just mean do one test - I refer to my anecdote above and testing ‘both sides of the coin’. Do a run of the configuration play, then do another run. Do both arrive at the same place - i.e. is the full service delivering what it is supposed to?

Bigger picture context is important - I want to stress that what is in my head isn’t just confined to testing a few plays. I imagine in a well thought out system, testing of infrastructure automation would be, well, automated. If I were building out another infra today I would have a CI job that took IaC definitions and spun up VMs to test the end to end, automatically. Why not have an overnight job that takes the day’s CM work and builds the application from the ground up? (then tests it is delivering what it should).

To me testing of individual roles is adding work, it’s minutiae that doesn’t matter - when the bigger picture is done correctly.

But, if it makes you feel safer about your infrastructure automation then why not? Go ahead and test every role - it’s perfectly possible to do. All I’m saying is I wouldn’t, I would put my effort into making sure the end goal is working correctly.

Because no matter how you get the ball down the pitch, the point of the game is still to get it through the goal posts.

Thanks for reading this post via RSS! 👏

Whilst I was giving a small pitch for Ansible, I fundamentally wanted to talk about simplicity (a common theme throughout this blog, you’ll find, way before I went to work for Ansible, Inc) and its role in enabling longevity to systems engineering.

As I’ve come back to this topic in late 2024, I’ve realised how this talk still carries relevance nearly a decade later. A short while ago I posted this on LinkedIn, and it got 10x my usual views. It clearly struck a chord:

The tech industry is a shambolic mess now (to cheer yourself up, go read any of Ed Zitron’s writing) — and so much of it is down to companies not being willing (or, perhaps, able) to put the work in to making things simple.

Simple doesn’t mean “basic”, it means the development of the software has had high quality engineering time and effort put into it (this also means it is not “easy”). It means there have been conversations between development and customer — conversations that have involved asking questions about the users’ day to day working, and listening. Then the hard work has been put in to remove any and all friction from those daily challenges.

If somebody can’t pick up, use and understand your software inside of 10 minutes then you’ve failed to put the work in. You have failed to make it simple.

It is that simple.

Here are the slides:

Hello! My name is … [8s pause] We'll come back to the significance of that 8 seconds later. #

Cue short video of Mark Adams. First 30 seconds: "A good system…" https://youtu.be/QlgXFRqfyWo #

Short bit about Vitsoe's shelving. Fully bespoke… #

it can serve any function you need… #

in any room… #

it can be small… #

or large. Did you think you were coming to a tech talk today? 😂 — Don't worry, we'll come to tech in a moment. #

But here's the beauty of the 606 shelving system. As bespoke as it is, it actually isn't. It's a system. A very simple system, of components that can be slotted together to make 'bespoke' installations. #

Does anyone know who this is? (when I delivered this talk in early 2015, barely anyone had heard of Dieter Rams). Rams was the head of design at Braun from 1961 to 1995, and designed the Vitsoe 606 shelving system in 1959. It is still fundamentally the same today, and Vitsoe is very proud to say customers take their furniture as they move house, remodelling the existing components as they do. #

I could relate to that (in no way comparing me to Rams! I am but an engineer). I was starting to despair of the tech industry. Overwrought, overcomplicated 'solutions'. Let's discuss four of Rams' principles of good design. #

I was pitching Ansible here, so talked about its simplicity, yet flexibility. Adding things on to it that were not a 'shipped' part was easy. I do believe if you can't achieve something within 15 minutes of first picking up a new product though, the product design has failed. #

YAML for configuration. Unlike JSON, and in the olden days, XML 😱 it's human readable. Makes it very accessible — even somebody who has never used the product before instantly gets it. This should be a fundamental part of any product design — it needs to be understandable, accessible, almost instantly. #

The ‘Unix philosophy’ - Ken Thompson's early meditations on how to design a small but capable operating system
Use older tech. It works. SQL, SSH, keep web interfaces as simple as possible. A lot to be said for server-side HTML where interaction is barely needed. Just look at the state of the web today [2025]. #

The most important observation that I have made about good design—it’s not about aesthetics. Good design focuses on people. We would do well, in the software industry, to pay attention to Rams's ten principles, and to remember that ultimately we're using technology to make our lives easier, or better. #

Ansible promise back in the day. I swapped the 2nd and 3rd columns around from the original order, because this flowed better. Almost like a magic trick — simple, but powerful, and the turn; it was agentless. Which for automation tools back then was unheard of. #

And the significance of the 8 second pause at the start? I didn't want the audience to take any of my words to convince them of anything. I wanted them to try the product… #

and showed it was possible to install it in 8 seconds. #

Wise words. We're still getting that so wrong even today, in 2024, sadly. The tech industry would do well to study, and adhere to, Rams' 10 principles of good design https://www.vitsoe.com/gb/about/good-design #

Thanks for reading this post via RSS! 👏

Ansible Core contains a module for managing virtual machines in VMware vSphere environments called vsphere_guest. Using this one module we can talk to an existing vSphere instance to create new VMs, clone VM templates, and control and delete VMs. Couple up a simple playbook with Ansible Tower and we can do some pretty amazing things with very little effort.

Here’s our playbook to create a new virtual machine from a template in vSphere:

- hosts: vmcreate
  gather_facts: false
  connection: local
  vars:
    vcenter_hostname: 10.10.10.1
    esxhost: 10.10.20.1
    datastore: NFS001
    network: VLAN001
    vmtemplate: CentOS7
    vmcluster: AppsDev
    notes: Created by Ansible
  tasks:
    - name: Check for required variables
      fail: msg="Must pass name to -e"
      when: name is not defined

    - name: Check for vSphere access parameters
      fail: msg="Must set vcenter_user and vcenter_pass in a Vault"
      when: (vcenter_user is not defined) or (vcenter_pass is not defined)

    - name: Create VM from template
      vsphere_guest:
        vcenter_hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_user }}"
        password: "{{ vcenter_pass }}"
        guest: "{{ name }}"
        from_template: yes
        template_src: "{{ vmtemplate }}"
        cluster: "{{ vmcluster  }}"
        resource_pool: "/Resources"
        vm_extra_config:
          notes: "{{ notes }}"
        esxi:
          datacenter: Dev
          hostname: "{{ esxhost }}"

The really important bit that’s doing the work is the 15 lines associated with the ‘Create VM from template’ task. Note the extensive use of variables in this play to give us lots of flexibility. We’ll make use of this in Tower in just a moment.

Tower 2.1 introduced ‘Surveys’, which are a great way to create interactive forms for a given play. The form can prompt for variable information to make running a flexible play simpler:

Setting up multiple choice questions is also really easy:

There are many more ways to extend this simple functionality - Tower’s job scheduler could automatically run a play to ‘scavenge’ unused VMs, for example (put a fact on the host to say it’s done with, and you protect against automatically killing a VM still in use).

(This post originally appeared on ansible.com as part of my then day job).

Thanks for reading this post via RSS! 👏

For my particular need, all I wanted was for newly booted hosts to automatically find their configuration. Since I am using Ansible in its push based model, this meant allowing my newly provisioned host to ‘phone home’ - announcing its presence so that my centralised Ansible host could push the configuration. My centralisation is controlled by Ansible Tower.

But what if a network is fluid? What if I want a ‘baked template’ (for example, a VMware template) without hard coded hostnames in? How would my new host work out where to phone home, or for that matter, how to phone home?

What we need is a reliable, tried and tested, centralised infrastructure that can be used as an information source. What could possibly fit that need? Oh, yes, how about DNS? And how would we make use of it for this purpose? With two simple records - an SRV and a TXT.

In the vein of keeping things simple, on this particular network I am using dnsmasq. It wouldn’t be much more complex to use BIND, of course.

I’ve created two records in dnsmasq, one to allow my hosts to discover where to phone home to, and a second to provide information on what to ask for. Here are those records:

srv-host=_cm._tcp.lan,tower.lan,80,0,0
srv-host=_cm._tcp.lan,tower2.lan,80,20,0
txt-record=lan,"59301625893930005661466df034870d 20"

Let me explain them.

There are two SRV records - you’ll notice the difference being an apparent hostname, and a number towards the end of the line. The line is described like so:

srv_record, hostname, port, priority, weight

Priority and weight allow us to have multiple records for resiliency, with a lower priority taking precedence, and higher weights of the same priority being more important. I tend to skip using weightings, and set the priority fields alone. So here, “tower.lan” takes priority over “tower2.lan”.

Secondly we have the TXT record, with the first portion being a hostname - or in this case, the whole domain, and the second portion being a free form amount of text. I’ve kept this pretty simple, because it works nicely for this next portion.

Ansible Tower has a really neat function called ‘callbacks’, which exposes a given job template as a REST URL. This then permits calling the URL with tools as simple as cURL. It isn’t intended for heavy use (i.e. it’s not there to implement a pseudo client/server architecture) but for ‘phoning home’ to get an initial configuration, it’s brilliant.

Poking the callback URL needs three things to be in place for it to work - a ‘belt and braces’ approach if you like. First, the calling host (our ‘client’) needs to be known to Tower’s inventory; "your name’s not down, you’re not coming in". The client then needs to poke the URL with two pieces of information - a “host config key”, which Tower generates for us, and a job ID, which we get from the template.

Shipped with Tower is a nice little wrapper around this call - look for request_tower_configuration.sh in /usr/share/awx.

Pulling all this together, I’ve included request_tower_configuration.sh as part of my VM template, and I simply run it at first boot like so:

request_tower_configuration.sh $(dig +short _cm._tcp.lan srv | awk '/^0/ {print $4}') $(dig +short lan txt | tr -d '"')

The two dig commands grab the information request_tower_configuration.sh needs - the host to contact and the config key and job ID.

Discovering information like this gives me the flexibility to move Tower hosts around, and even to change the configuration job that’s requested; without having to do anything to my baked template. I can even offer different jobs to hosts with different DNS views - using a more sophisticated DNS server like BIND could give us quite a bit more flexibility. You could take the TXT record theory even further - using different records to provide different actions.

Fundamentally what this has given me is a lot of flexibility without having to ‘reinvent the wheel’.

Lastly, the keen eyed among you will notice my ‘cheap’ call here ignores my SRV record resiliency. Yes, indeed it does. I leave that as an exercise to the reader. You might like to write your own wrapper around cURL, for example.

Thanks for reading this post via RSS! 👏

1. Rule of Modularity: Write simple parts connected by clean interfaces.
2. Rule of Clarity: Clarity is better than cleverness.
3. Rule of Composition: Design programs to be connected to other programs.
4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
5. Rule of Simplicity: Design for simplicity; add complexity only where you must.
6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
7. Rule of Transparency: Design for visibility to make inspection and debugging easier.
8. Rule of Robustness: Robustness is the child of transparency and simplicity.
9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.
10. Rule of Least Surprise: In interface design, always do the least surprising thing.
11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.
12. Rule of Repair: When you must fail, fail noisily and as soon as possible.
13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
16. Rule of Diversity: Distrust all claims for “one true way”.
17. Rule of Extensibility: Design for the future, because it will be here sooner than you think.

Thanks for reading this post via RSS! 👏

From presenting at DevOps Cardiff and the London Splunk User Group, I have tried to spread the word. Because Ansible is the embodiment of something I’ve been going on about for years - the tech industry keeps trying to make things more complicated than they need to be.

Making things complicated isn’t the answer, making them simple is!

So I am proud to say that I’m becoming a small part of Ansible, Inc. From today, I will be their first representative in the UK and Europe.

For the short term, I’ll still be juggling my commitments with VNTX and DevOpsGuys, but you can also reach me as mark AT ansible.com

Thanks for reading this post via RSS! 👏

I’ve been working in IT for just over 20 years - I was very lucky to ride the greatest wave we’ve seen, the dawn of the Internet (I worked for the largest corporate ISP in the world, UUNET, from early 1996 to 2001), and I’ve worked in the slowest, most immobile companies you can imagine (Investment Banks). And in the last 5-10 years I’ve seen less and less common sense be applied. And now we have this word that nobody can truly define. And it’s creating even more silliness. People are spending energy, lots of energy, debating this phrase and its definition (the irony of this post is not lost on me).

I despair of Agile as a formulaic solution to all our woes, I really do. I watch organisations debate over Kanban versus Scrum, then pick one and apply every last facet of the methodology to everything they do - in effect making project management so granular as to become weighty and patronising micromanagement. Yes, there are great points to it - releasing often is better than spending months, years, planning something then trying to release in a big bang. But at the same time barely planning past two weeks is akin to driving along the road looking at the end of your car bonnet.

I feel sorry for those organisations that pick up yet another piece of software that they believe will ‘fix everything’ for them, but in reality they purchase a well sold lump of complexity that could be achieved much, much simpler.

Then Mike at UpGuard brought to my attention an article they’d posted about DevOps. And I think it’s spot on, in a single paragraph:

“The core of DevOps is not a difficult or complex concept. More or less, it’s the idea that developers and operations should collaborate more closely than they traditionally have, so neither side is caught with their pants down when something blows up”

The core ethos has spawned many, valid, accoutrements, but at its simplest definition it is about us all ‘rowing in the same direction’. Developers, Operations, we’re all in the same boat. Let’s work together to make it go forward. Creating a ‘DevOps Team’ is completely missing the point - you’ve just created yet another silo, where if the culture isn’t worked at (like any solid relationship, a culture of communicating and sharing needs constant work), communications will fail and we’ll have three teams doing their own thing.

Across my working years I’ve been a network engineer, a systems engineer, a network manager, a project manager, a security consultant and a technical consultant. Every role has involved working with other people to deliver a business goal. And I’ve always found that the goal was much easier to hit when I talked to my colleagues and customers.

It’s not difficult, it’s common sense.

But what is it they say about common sense? It’s surprising how uncommon it is.

Thanks for reading this post via RSS! 👏

The easiest way for me to achieve this was to send the logwatch output to a file on a NFS volume.

However, the host in question has SELinux enabled (quite deliberately). Often the answer to SELinux problems is ‘disable it’, but that’s not the proper answer! And really, I could do with learning it.

First off, analysis of the audit log told me what I suspected - it was SELinux preventing the write:

# ausearch -m avc --start today 
----
time->Sun Jan 12 03:14:05 2014
type=SYSCALL msg=audit(1389496445.113:2441932): arch=c000003e syscall=2 success=no exit=-13 a0=2817680 a1=441 a2=1b6 a3=37ba71dd40 items=0 ppid=32511 pid=32517 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=31998 comm="logwatch" exe="/usr/bin/perl" subj=system_u:system_r:logwatch_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1389496445.113:2441932): avc:  denied  { write } for  pid=32517 comm="logwatch" name="host" dev=0:1c ino=1834908 scontext=system_u:system_r:logwatch_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nfs_t:s0 tclass=dir

The key bits to note here are the scontext, the tcontext and the tclass - source and target contexts, and the target class.

Essentially, the target directory was of ’type’ nfs_t, and the source process was of logwatch_t type.

Let’s check the default SELinux policy for this combination:

# sesearch -t nfs_t -s logwatch_t -c dir -A
   allow logwatch_t nfs_t : dir { ioctl read getattr lock search open } ; 

Ah - so logwatch_t cannot write to a dir of type nfs_t. That’ll be our problem.

We can fix this by writing a custom module. The easy way is to take our denied log entry and pipe it through audit2allow. First off, let’s do that, but output to a text file for review, before we implement the rule:

# ausearch -m avc --start today | audit2allow -m logwatch_nfs_write -o logwatch_nfs_write.te

(we could just have easily grep’d for the offending line in the actual audit.log file)

Now review the output file to make sure it’s all sensible:

module logwatch_nfs_write 1.0;
 
require {
        type logwatch_t;
        type nfs_t;
        class dir write;
}
 
#============= logwatch_t ==============
#!!!! The source type 'logwatch_t' can write to a 'dir' of the following types:
# var_lock_t, tmp_t, logwatch_cache_t, logwatch_tmp_t, var_run_t
 
allow logwatch_t nfs_t:dir write;

OK, looks sensible. So now we can compile it, and insert into the running modules:

# ausearch -m avc --start today | audit2allow -M logwatch_nfs_write
# semodule -i logwatch_nfs_write.pp

And check it’s in memory:

# semodule -l | grep logwa
logwatch_nfs_write      1.0

Now when I run logwatch the file is created. Sorted.

There are some examples in /usr/share/selinux/devel, on CentOS at least, that could be useful for writing modules from scratch.

Thanks for reading this post via RSS! 👏

Although I wasn’t actually trying to install Jenkins, my workflow led me to look at installing it with Ansible as a proof of concept. Basically I wanted to see how easy this task could really be.

I’ve put the playbook, along with some notes, into a Github repo, which you should feel free to fork and play around with.

Currently this is a very simple example - spin up the EC2 instance, then install and run Jenkins as its own daemon (on the default port of 8080).

The playbook includes an example of how to use Ansible’s Route53 module. See the readme for extra notes.

Enjoy.

Thanks for reading this post via RSS! 👏

There are many things that could be given the label of “the greatest piece of engineering”, and we could debate them forever. I’m sticking to my statement. But why?

Because, in over 100 years since The Safety Bicycle came about, the ‘double diamond’ frame design has basically gone unchanged¹. The fundamental design cannot be improved on.

That, to me, is quite some achievement. And the beauty of it is not complexity. It is simplicity.

How does this relate to technology? I’ve mentioned before that my favourite quote is from Einstein, “Make everything as simple as possible, but not simpler”. I am also a fan of Dieter Rams’ 10 principles of good design.

Rams’ last point echoes the Einstein quote - “Good design is as little design as possible”. “Less, but better – because it concentrates on the essential aspects, and the products are not burdened with non-essentials. Back to purity, back to simplicity”.

It is possible to build quality IT infrastructures these days without resorting to complex and bloated products that promise an easy life. Supporting a complex environment is considerably harder, it takes longer for new hires to learn, and time taken to resolve problems is drastically extended. These translate to higher costs to a business, and a very real reduction in the ability to meet customer expectations. What might have seemed a quick shortcut to take initially has led you into a dense and overgrown forest.

The phrase du jour, DevOps, has had much interpretation. To me, it is a label applied to what any business with quality engineering would have been doing for years anyway. Plus, it is more about culture than technology. Giving developers a smooth route to production needn’t be a minefield of complexity, wrapped in layers and layers of ’trendy’ software products that are doing a simple job.

Returning to the bicycle, here’s The Hovenring in the Netherlands:

The challenge was to keep traffic, of all sorts, moving in a smooth and safe way. They could have introduced phased traffic lights, or burrowed under the ground; but both of these solutions have obvious negatives. There is smart engineering at work here - something as simple as it needs to be.

The DevOps integration of developers and operations isn’t just about technology, it is about communication. It is about people. It should be a consultative workflow, where the right skills are used to design the solution. It isn’t about taking yet another product and throwing it into the infrastructure soup, it’s about discussing business goals and using both sets of skills to achieve them.

Regardless of the label, progressing a business’s technology can be achieved through smart, simple, engineering without being overwrought.

I’ll finish off with a link to a short video. The Vitsoe shelving system, designed by Rams in 1959 and still fundamentally the same today, epitomises my thoughts on IT infrastructure design. The first 40 seconds are golden words we should strive for in the tech industry too.

Vitsoe ‘A persistent system’

Thanks for reading this post via RSS! 👏

Configuration Management is the epitome of automation, it forms the bedrock of achieving supportable, repeatable and consistent services. CM is also key to the phrase-du-jour, ‘DevOps’. Your end to end (dev to prod) application deployment is nowhere without CM.

This is an opinion piece. There are no tests, they’re simply observations with comments. Therefore it’s very subjective. You may well have a different point of view. I welcome comments.

Some salient points:

1. I’ve been working with Puppet as my choice of CM for over six years. I’ve implemented it for a good few clients, and worked with it in other, established, infrastructures. And for the most part, I like it.
2. Despite that, I’m not the biggest fan of Ruby. Sure, I write some stuff in it, but it wouldn’t be, isn’t, my first choice of ‘glue’ language.
3. I’m a UNIX person. I don’t care about anything from, or relating to, Microsoft.
4. I am pragmatic. Whereas I like to do things the best way possible [that I think ;-)] I will also implement with the client in mind. For the most part I am software agnostic (apart from having favourites, of course).
5. My favourite quote is from Einstein: “Make everything as simple as possible, but not simpler”.

So, let’s get on with it.

Puppet

I’ve already mentioned that I’ve a solid background with Puppet. The main thing I like about it, and always have done, is the DSL. Puppet has a nice, simple, language for describing your systems. “Package”, “Service”, “User”, “requires”, etc. etc. It’s simple to read, easy to understand, and system agnostic.

It’s a fairly trivial install with few dependencies [Ruby], and if you use the Enterprise version it’s even easier. And unless you’re using Dashboard, you don’t really need a database (which for small implementations - of anything - is good in my book).

A lot of large organisations want a GUI, and want to be able to manage systems from it (be that web or otherwise) - which Puppet Enterprise provides (you can run Dashboard as open source too). And, although I don’t care in the slightest, it supports Microsoft Windows too.

Puppet themselves maintain package repos that contain most of the dependencies too (there are some Ruby modules that aren’t in RedHat base distros, like CentOS, for example), which again makes life a bit easier.

There’s a great community, and lots of modules have already been written ’to do stuff’. Their founder, Luke Kanies, wanted a solution to stop sysadmins ’re-inventing the wheel’ at every company. I think Puppet achieves that.

The negatives - well, sometimes you can get into a pickle with resource dependencies (e.g. service X needs package Y and config file Z). You can only define a ‘resource’ in one place (e.g. an httpd service), which for newbies can be confusing. But I’m nit-picking, you get over that one quick enough.

It can be slow when you have hundreds or more clients trying to get their catalog. I’ve given up running Puppet as a client-server architecture now, and run ‘masterless’. This seems to be an increasing trend. Put all your manifests into a version control system (say, git) and check that out on every host. You’ve a simple, trusted, transport mechanism in SSH, and the application of the configuration just runs locally. No overloaded central server to worry about. Run the checkout under cron, with a random timing element, and you don’t generate a different performance problem by every host SSH’ing in at the same time.

Chef

I’ve only recently been exposed to, and had to use, Chef. I didn’t chose to use it, but being pragmatic I went into the piece of client work open minded - I knew Chef came about shortly after Puppet, which influenced its formation. I had heard they were similar products.

But for starters I’ve been put off by its DSL - it’s Ruby. Just pure old Ruby. I guess this would make it feel like ‘a comfortable pair of shoes’ for a Ruby developer to start managing systems, but coming from a systems background all I can see is it makes it hard work for anybody not a programmer by trade. It also seems overly complex. Workstation, Server and Clients. Knife, chef-client, chef-solo. Data structures in JSON. A Postgres database.

I’m also seeing a lot of ‘shelling out’ in the manifests at this particular client, which makes me think they were unable to achieve what they wanted within Chef too (’execs’ are frowned upon in the Puppet community - but of course, if you’re just running shell scripts on a system it is rather bloated and inefficient to be wrapping them in Ruby! You might as well go back to old skool for-looped copying of scripts over SSH).

And this is the thing. If you are a web programmer, maybe working on Rails, then all this is familiar ground, so it seems simple to pick it to manage your systems.

But, wait, why would you? Why would you pick something that is, as far as I can see, Puppet made overly complex? Nothing about Chef makes sense to me when I compare it to other CM solutions.

Even when you factor in somebody coming at it from a familiar background, it wouldn’t take a Ruby programmer, or any programmer for that matter, at all long to ‘get’ Puppet’s DSL, it is that simple. Puppet has a data store like Chef’s data-bags, but it’s YAML instead of JSON. And when I look at the two file formats, I fail to see why you would use JSON (for a human readable data store, I mean. Granted it’s a web programming standard, so discount that) when YAML is so much more readable.

If I’d come to Chef before Puppet I’d probably see all its good points - it delivers centralised configuration management in a, relatively, [Ruby] easy way. But I’ve not, I’ve used something that is similar but oh so much simpler.

I just cannot see the point of picking Chef over Puppet.

UPDATE: Interestingly, in the comments below, somebody brought to my attention an old blog post elsewhere that said of Chef - “It’s a remote execution engine with a great deal of architectural overhead”.

But wait, we’re not done yet…

Ansible

Ansible is, again, another recent exposure for me. At pretty much the same time as Chef, since it’s being used as part of the same client project - which begs the question, why didn’t they just stick to one CM solution?

Well, because it was being used to CM the setup of XenServers. Which don’t have Ruby on. And Ansible is based on Python, which is on pretty much any base install of an OS these days (most of RedHat’s toolset on RHEL is written in Python, and even Sun were fond of writing tools in it too).

And this is where it got interesting for me. Ansible is the work of Michael DeHaan. Some of you may know about Cobbler. When I learned this I was instantly put off without even trying it (talk about judging a book by its cover) - I never liked Cobbler. Too complicated for something that should be simple (this is one of the reasons I wrote my own - totally not worthy of wider use! - Linux kickstart tool, Bacio).

But then I started to use Ansible. And my first impression was good. Here is a tool that works opposite to the way Chef and Puppet do [client-server]; it’s a push model. It requires no additional installs on the end points (unless you’re using selinux, then it takes one extra Python library that’s more than likely in the distribution repository anyway). It uses SSH for its transport. And YAML (which I really like because it meets my requirement of ‘simple as possible, but no simpler’) for the configuration language.

As I started to try to achieve things with Ansible I found it to be the simplicity I want to see from technology. It could achieve all that I’ve done with Puppet for years, but in an even easier way.

The YAML ‘playbooks’ fall through tasks like running down a firewall ruleset. You can see the path your job takes right before you (Puppet does a complex catalog compile to work out ordering, which can change from run to run. This has positive and negative connotations, not least some occasional confusion).

No remote installs. No funny transports and odd ports to have to open on firewalls. No complex SSL certificate management. No database servers.

And I’m increasingly realising DeHaan (and one presumes his co-developers) have thought of most things. You could run it client-server if you chose; by simply emulating the master/server-less models of Puppet & Chef[-Solo].

They have a commercial offering, Ansible Tower, should you want a pretty dashboard. It’s actually got some really nice features too, one of my favourite being the REST API and Callbacks (see page 114 of the Tower manual for more information)

Ansible. Oh how I wish I’d not been introduced to Ansible. Because now I can’t think about doing CM with anything else. I could still proceed with Puppet, (if I really had to) but Ansible is at the other end of the spectrum to Chef as far as I’m concerned.

I’m sure there’ll be negatives somewhere along the line, there usually are with technology. I just haven’t found them yet. The obvious one I don’t care about - it doesn’t support Windows[3] (really? You wouldn’t train for a marathon by riding a bicycle, you’d run. So why try to lump management of Windows and UNIX/Linux into the same space? Hire people who do Windows, hire people who do UNIX, and let them manage their own estates).

You’d probably have to do some funky routing and firewalling in a massive estate to use the push model (you’d probably opt for the client-server mimic). And in its current form managing a large estate would probably be a bit like hard work. But for an SME, or a startup, managing an estate from the shell? I think it’s brilliant.

Summary

The long and short of it:

• Ansible works really well. I can’t see the negatives¹. It is my choice of CM software right now, and I will encourage clients to use it whenever I can.
• If you’re absolutely against Python, SSH, and particularly want to install agents on every machine you manage, or you need to manage Windows² right now, pick Puppet.
• I can’t actually think of a good reason to use Chef. Even if you’re a Ruby programmer you’ll still be better off with Puppet.

[3] Update, June 19th 2014; Ansible now supports Windows. Or, at least, is beginning to. See this blog post.

Thanks for reading this post via RSS! 👏

You’ll need to turn on logging of your bucket in S3.

Next, install the S3 app for Splunk. Configure an S3 input (Manager >> Data inputs >> S3). Your bucket logs will now automatically feed into Splunk (the sourcetype will be ‘s3’).

I’ve written a ‘rex’ filter to grab the fields I want to work with - date, IP, action (REST) and file[name].

Stick this into the search field, then you can do all the usual Splunk goodness with the resulting fields…

sourcetype="s3" | rex field=_raw "(?x: \[ (?<date> .*) \] \s (?<ip> (?:\d{1,3}\.){3} \d{1,3}) .+? REST\.(?<action> \w+)\.OBJECT \s (?<file> .+?) \s)"

Thanks for reading this post via RSS! 👏

The problem for me was if I ran SSL, the redirect was talking plain http, and would try to redirect to the remote Splunk using http. My Splunk instance is also running on a different machine to the Nginx front end.

So here are the config file details:

nginx conf.d/splunk.conf

server {
    listen 443 ssl;
    server_name log log.domain.com;

    access_log  /var/log/nginx/splunk_access.log;
    error_log   /var/log/nginx/splunk_error.log;

    ssl                       on;
    ssl_certificate           certs/my.crt;
    ssl_certificate_key       certs/my.key;
    ssl_session_timeout       5m;
    ssl_ciphers               HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache         shared:SSL:128m;

    location / {
        auth_basic           "Login";
        auth_basic_user_file auth/users_file;

        proxy_pass       https://192.168.1.1:8000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

splunk $SPLUNK_HOME/etc/system/local/web.conf

[settings]
enableSplunkWebSSL = 1

Thanks for reading this post via RSS! 👏

However, using Modern::Perl means your perlcritic check will fail with ‘code before strictures’. I also use Syntastic in Vim, which means every bit of Perl I write these days gets a nag.

I would’ve thought Perl::Critic would catch up, but evidently not. Then I discovered it’s actually pretty simple to accept use Modern::Perl as valid code:

Marks-MacBook:~$ cat ~/.perlcriticrc 
[TestingAndDebugging::RequireUseStrict]
equivalent_modules = Modern::Perl
 
[TestingAndDebugging::RequireUseWarnings]
equivalent_modules = Modern::Perl

Sorted.

Thanks for reading this post via RSS! 👏

As rough as it stands today, a question prompted me to publish the helper I use. So here you go, my Perl Mojolicious helper, Bacio.

Thanks for reading this post via RSS! 👏

UPDATE in 2024: Hello! This blog post is over 10 years old now, yet here we are in 2024 and I’m seeing a large uptick in traffic coming here to read this. So before you read on, could I ask a small favour please? I’m really intrigued as to why traffic has increased, so would you mind dropping me a quick email via the link below, or pinging me on Mastodon or LinkedIn, letting me know what you were looking for, and maybe a little about the problem you’re trying to solve? The reason why I find this so intriguing is, the traffic spike tells me there’s some sort of shift in infrastructure going on; maybe people are bringing workloads back from the cloud more frequently? Anyway, I’d love to hear from you as to how you ended up here at this post. Thanks for taking time to read this bit, please do continue to read what you came here for 😊

Sometime ago I looked at iPXE. There is a nice capability with it to add a script, or link, to the initial boot process. Using this I rolled a single ISO (you could put it in a DHCP/PXE/TFTP setup - I did it this way because I wanted a simple, transportable, solution to use on my desktop and laptop) with a link to a CGI I’d written. Using the CGI I do a lookup, based on MAC address, for the host’s intended Linux version and a tailored kickstart file.

One ISO can therefore boot multiple Linux versions and even host-specific kickstart files (which are always kept to the most bare bones they can be - handing off to a configuration management tool like Ansible or Puppet in the %post section to let it handle what it does best).

Here’s an example file that is rolled into the ipxe.iso build with make EMBED=menu.ipxe bin/ipxe.iso

host:src$ cat menu.ipxe
 #!ipxe
 dhcp net0
 chain http://192.168.1.1:3000/pxe/${net0/mac}

And an example of the output from the CGI:

MacBook-Pro:~$ curl -s http://192.168.1.1:3000/pxe/08:00:27:59:86:69
#!ipxe
kernel -n img http://build/centos/6.4/os/x86_64pxeboot/vmlinuz ks=http://192.168.1.1:3000/ks kssendmac
initrd http://build/centos/6.4/os/x86_64pxeboot/initrd.img
boot img

The pretty rough CGI I use is written in Perl Mojolicious, and can be found on Github.

Thanks for reading this post via RSS! 👏

Lots and lots of solutions for this one out there in Google land, but it turned out to be quite simple.

This is using Perl 5.14.3 on Linux, with LWP 6.04 using IO::Socket::SSL 1.77

I’d set the client up like so:

my $ua = LWP::UserAgent->new;
$ua->ssl_opts(
    SSL_ca_file => "$ENV{'HOME'}/ca.pem",
    SSL_cert_file => "$ENV{'HOME'}/my.crt",
    verify_hostname => 1,
);

…my crt file is a combined certificate and key.

The solution turned out to be simply a case of including the key statement, pointing at the same file:

$ua->ssl_opts(
    SSL_use_cert => 1,
    SSL_ca_file => "$ENV{'HOME'}/ca.pem",
    SSL_cert_file => "$ENV{'HOME'}/my.crt",
    SSL_key_file => "$ENV{'HOME'}/my.crt",
    verify_hostname => 1,
);

Thanks for reading this post via RSS! 👏

Recently I completed a project for a client using Mojo::UserAgent as a server to write a simple monitoring dashboard. But this week I’ve been using the command line ‘mojo’ tool to grab some stock prices from the web. And it’s brilliant how simple it is.

Here’s how to grab AAPL from Google Finance, outputting the current price:

mojo get https://www.google.com/finance/quote/SPY:NYSEARCA ".fxKbKc" text

Working out the CSS selectors is simply a case of looking at the web page in your browser using the debug tools to look through the source. I’ve also found this resource on CSS selectors handy, as I don’t do this every day I’m prone to forgetting the syntax.

Nearly 11 years later, in 2023, I discovered xq 🤣 You know what’s even simpler than mojo? This:

curl -L https://www.google.com/finance/quote/SPY:NYSEARCA | xq -q ".fxKbKc"

Thanks for reading this post via RSS! 👏

Server version unavailable at '<a href="https://hostname/sdk/vimService.wsdl">https://hostname:443/sdk/vimService.wsdl</a>' at /usr/lib/perl5/5.8.8/VMware/VICommon.pm line 545.

Turns out it can’t verify the server’s SSL certificate.

export PERL_LWP_SSL_VERIFY_HOSTNAME=0 in your shell before using a command, and it then works correctly.

Thanks for reading this post via RSS! 👏

When it comes to extra Perl modules, I always install everything in my home directory, using local::lib - it keeps everything neat and tidy and means I don’t need to install anything under sudo.

However, I’ve just been struggling to get DBD::mysql to work properly with the packaged MySQL. It would compile just fine, but then refused to load for testing, seemingly unable to find the libmysqlclient dynamic library:

#   Failed test 'use DBD::mysql;'
#   at t/00base.t line 21.
#     Tried to use 'DBD::mysql'.
#     Error:  Can't load '/Users/markp/.cpan/build/DBD-mysql-4.018-hYHpKY/blib/arch/auto/DBD/mysql/mysql.bundle' for module DBD::mysql:
dlopen(/Users/markp/.cpan/build/DBD-mysql-4.018-hYHpKY/blib/arch/auto/DBD/mysql/mysql.bundle, 2): Library not loaded:
libmysqlclient.16.dylib

After a bit of Googling around, various resources led me to the fact the compiled binary was lacking an RPATH. So all I needed to do was link the binary with an RPATH. But it seems things are not as simple on OS X.

According to Joe Di Pol’s blog, OS X works slightly backwards to what most of us from Linux and Solaris backgrounds understand - compiled binaries look at a dynamic library, which in turns says where it is, rather than the traditional way of thinking which is to include a library search path in the compiled binary.

To see what I mean, look at the mysqlclient dynamic library with otool:

DBD-mysql-4.018-hYHpKY$ otool -D `mdfind libmysqlclient.16.dylib`
/usr/local/mysql-5.5.9-osx10.6-x86_64/lib/libmysqlclient.16.dylib: libmysqlclient.16.dylib

There is no path listed there, which means binaries compiled to use the library won’t be able to find it unless it’s in the system search path of /usr/lib.

We can fix this with install_name_tool though:

DBD-mysql-4.018-hYHpKY$ sudo install_name_tool -id /usr/local/mysql-5.5.9-osx10.6-x86_64/lib/libmysqlclient.16.dylib
/usr/local/mysql-5.5.9-osx10.6-x86_64/lib/libmysqlclient.16.dylib

DBD-mysql-4.018-hYHpKY$ otool -D `mdfind libmysqlclient.16.dylib`
/usr/local/mysql-5.5.9-osx10.6-x86_64/lib/libmysqlclient.16.dylib:
/usr/local/mysql-5.5.9-osx10.6-x86_64/lib/libmysqlclient.16.dylib

Now when we run the tests:

DBD-mysql-4.018-hYHpKY$ make testPERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')"
t/*.t
t/00base.t .................. ok
t/10connect.t ............... ok

It works!

There may be a better way to do this, and if I find it I’ll update this post. Otherwise, that works just fine for me.

Thanks for reading this post via RSS! 👏

The idea was to patch a Solaris 10 update 8 (10/09) machine to the most recent patch levels, whilst the machine was still up and running, going about its daily business. Other than the standard Solaris tools, I’d be using pca Patch Check Advanced to do the actual patching. The system was installed with a ZFS root, since this actually gets us some great features in LiveUpgrade (LU) - namely ZFS snapshots as boot environments (BEs).

First off, create a BE that will be patched:

solaris:~# lucreate -n patching
Checking GRUB menu...
System has findroot enabled GRUB
Analyzing system configuration.
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment .
Source boot environment is .
Creating boot environment .
Cloning file systems from boot environment to create boot environment .
Creating snapshot for on .
Creating clone for on .
Setting canmount=noauto for in zone on .
mount point options <-> from parent filesystem file
type <-> because the two file systems have different types.
Saving existing file in top level dataset for BE as //boot/grub/menu.lst.prev.
File propagation successful
Copied GRUB menu from PBE to ABE
No entry for BE in GRUB menu
Population of boot environment successful.
Creation of boot environment successful.

If we now take a look at the ZFS filesystems we can see the ‘patching’ snapshot…

NAME USED AVAIL REFER MOUNTPOINT
rpool 2.08G 5.73G 33K /rpool
rpool/ROOT 1.03G 5.73G 21K legacy
rpool/ROOT/install 1.03G 5.73G 1.03G /
rpool/ROOT/install@patching 59.5K - 1.03G -
rpool/ROOT/patching 120K 5.73G 1.03G /
rpool/dump 560M 5.73G 560M -
rpool/export 44K 5.73G 23K /export
rpool/export/home 21K 5.73G 21K /export/home
rpool/swap 512M 6.14G 100M -```
Let's see what lustatus shows us now...
```solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes yes yes no -
patching yes no no yes -

So we have two boot environments.

LU has a nice feature of letting you mount a BE to do ‘work’ on it. Let’s see what’s mounted, and then mount our newly created ‘patching’ BE…

install on /
solaris:~# lumount patching
/.alt.patching
solaris:~# lumount
install on /
patching on /.alt.patching

So now our alternate boot environment is mounted as /.alt.patching, we can go ahead and patch it. pca supports patching to an alternative root with the -R switch, much like Solaris packaging tools…

solaris:~# pca -i -R /.alt.patching
[snip]
------------------------------------------------------------------------------
141505 04 < 07 RS- 28 SunOS 5.10_x86: ipf patch
Looking for 141505-07 (29/84)
Trying SunSolve
Trying https://sunsolve.sun.com/ (1/1)
Done
Installing 141505-07 (29/84)
Unzipping patch
Running patchadd
Done
Reboot recommended
------------------------------------------------------------------------------
[snip]
------------------------------------------------------------------------------
Download Summary: 84 total, 84 successful, 0 skipped, 0 failed
Install Summary : 84 total, 84 successful, 0 skipped, 0 failed

This could take a while.

When the patching is complete, unmount the BE and set it to be the active one on the next reboot…

solaris:~# luumount patching
solaris:~# lumount
install on /
solaris:~# luactivate patching
System has findroot enabled GRUB
Generating boot-sign, partition and slice information for PBE
Saving existing file in top level dataset for BE as //etc/bootsign.prev.
A Live Upgrade Sync operation will be performed on startup of boot environment.
Generating boot-sign for ABE Saving existing file in top level dataset for BE as //etc/bootsign.prev.
Generating partition and slice information for ABE Copied boot menu from top level dataset.
Generating multiboot menu entries for PBE.
Generating multiboot menu entries for ABE.
Disabling splashimage
Re-enabling splashimage
No more bootadm entries. Deletion of bootadm entries is complete.
GRUB menu default setting is unaffected
Done eliding bootadm entries.
**********************************************************************
The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.
**********************************************************************
In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:
1. Boot from Solaris failsafe or boot in single user mode from the Solaris
Install CD or Network.
2. Mount the Parent boot environment root slice to some directory (like /mnt). You can use the following command to mount:
mount -Fzfs /dev/dsk/c0d0s0 /mnt
3. Run utility with out any arguments from the Parent boot
environment root slice, as shown below:
/mnt/sbin/luactivate
4. luactivate, activates the previous working boot environment and
indicates the result.
5. Exit Single User mode and reboot the machine.
**********************************************************************
Modifying boot archive service
Propagating findroot GRUB for menu conversion.
File propagation successful
File propagation successful
File propagation successful
File propagation successful
Deleting stale GRUB loader from all BEs.
File deletion successful
File deletion successful
File deletion successful
Activation of boot environment successful. 

Notice the message about what to do to recover the old session should the boot fail. Personally I keep a copy of that notice to hand, just in case.

So if we now look at lustatus, we can see our patching BE is the active on reboot…

solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes yes no no -
patching yes no yes no -

So let’s go ahead and reboot at a time that suits us. When the system comes back up we can see ‘patching’ is now the active BE…

solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes no no yes -
patching yes yes yes no -

And pca shows us there are no patches to be applied, so we’re up to date…

Using /var/tmp/patchdiag.xref from Mar/02/10

zfs list shows us that the patching snapshot is now using up space too…

solaris:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.82G 5.00G 36.5K /rpool
rpool/ROOT 1.77G 5.00G 21K legacy
rpool/ROOT/install 31.3M 5.00G 1.05G /
rpool/ROOT/patching 1.74G 5.00G 1.37G /
rpool/ROOT/patching@patching 377M - 1.03G -
rpool/dump 560M 5.00G 560M -
rpool/export 44K 5.00G 23K /export
rpool/export/home 21K 5.00G 21K /export/home
rpool/swap 512M 5.40G 100M -

Using a recent Solaris 10, with ZFS root, LU and pca gives us a very realistic way of patching systems in a working, production, environment without the pain of downtime and with a workable roll back strategy.

Thanks for reading this post via RSS! 👏

If the scheduled time is missed there is no way to change the post to ‘publish immediately’ from scheduled. This leaves the post as scheduled, and won’t actually re-post it no matter what you do (including setting a future time).

A quick fix is to update the post directly in the database.

Find the post you want (make sure it’s the parent post, anything with a post_status of ‘inherit’ is a ‘child’ post, or update to the original, you want the first post of its kind with a post_status of ‘future’) by selecting on the post_title, and once you’ve got the ID of the post, simply set the post_status to ‘publish’ and it will appear at the time you originally set it to be scheduled for. For example…

update wp_posts set post_status='publish' where ID='8';

Thanks for reading this post via RSS! 👏

Today I discovered a fix - although I don’t know the reason why.

I had a fairly restrictive umask set, system wide, in /etc/profile - 027. When I did ‘zfs set mountpoint=blah’ with that umask I wouldn’t be able to rm -rf successfully. However, if I moved the mountpoint, set umask to 022, then put it back where it should be, rm -rf works fine!

The underlying directory was left when I temporarily shifted the home directory, and it was owned root:root and 750, as the umask would suggest, but why that would then effect the ZFS mount on top of it (which was correctly owned by my own user ID) I do not know.

Anyone know why this would have been?

Thanks for reading this post via RSS! 👏