Interesting times in the world of software

About a decade ago, I was at a conference and talking to a fellow developer (I still call myself one, even though I don’t code so much these days) when he giddily told me about the funding he’d got for building a new piece of software he was hoping would make it big. It was a two year project and he’d got £100k funding. I asked if it was just him… and no, he had a colleague. So £100k, for two people, for two years? £100k didn’t sound a lot… £25k/yr each, basically. Or what you can earn in a much simpler tech support role. I decided not to say anything and leave the poor guy in peace, although this sort of work seemed a lot like gambling to me.

Today, things are different although there’s still a sniff of gamble about it overall. If you’re a developer it’s relatively easy to find a highly capitalised employer that’s positively dripping with money who will pay you £60k-£90k a year. Potentially quite a bit more. This reminds me of the late nineties dotcom boom. In 1997 I myself quit my safe but somewhat dull job at a multinational to become a freelancer, doubling my income almost immediately, and quadrupling it another year later. The new work was, in some ways, more interesting. It was also a lot more stressful, bad for my health, and definitely wasn’t the most exciting coding work. But it paid. I honestly don’t blame developers who decide to do what I did 25 years ago. It set me up. I think it was also a large part of why I had a heart attack in 2019… living out of hotels for a decade wasn’t healthy, and cheese became far too much a food staple for me as a vegetarian. However, the money was very good and it helped set me up. When you’re poor, it’s very hard to catch up and a good income was necessary for a while.

I bring this up because today I’m not ‘just a developer’ but actually run a web development company that specialises in websites and custom software for clients. And things are happening today that are reminiscent of the dotcom boom on the late nineties. 25 years have passed, but people don’t really change nearly as much as you may think.

The dotcom & Millennium Bug era

The late nineties were a period of post-recession growth and capital release. Banks had been deregulated, money was being created in the way it can be, and we were riding high on increasing productivity. Life felt good. And when money is created it can be invested.

There’s only one little problem in that. Sometimes, people get giddy and start splashing the money out too readily. The boom of the late nineties and early noughties, and the deregulation that encouraged it around the world, eventually led to the financial crisis of 2008. I’m a bit of a cautious soul, so even though I had plenty of income, I resisted borrowing too much to get a bigger house. In some ways I was foolish, because I could now be living mortgage free in the house I have now. But I figured that not having a big mortgage would afford me some other freedoms and I could use my money elsewhere. Mostly I just invested my money in solid companies. Friends, however, were telling me to invest in dotcoms. But I looked at the fundamentals. One example was a firm called Vocalis. They did, basically, telephone voice services software. Small team, and had some crazy valuation that was effectively equivalent of £20m per member of the staff. I rightly reckoned that was mad. My friend went ahead and pumped money in, and I mocked him. For a while I looked a fool. The value of the shares rose and rose.

Right now, there are loads of speculation bubbles. At the café at work I was trying to explain Bitcoin’s fundamental problems to our barista, when our receptionist came over excitedly wanting to know more. Both seemed interested in getting involved. That means the crash is likely imminent. They’re both lovely people, but in the economic chain, they’re nowhere near the top, which means that the speculation bubble is reaching it’s limits.

“If shoe shine boys are giving stock tips, then it’s time to get out of the market.” – Joe Kennedy, 1929 as the stock market was about to crash and lead to the Great Depression

So the dotcom boom and Millennium Bug led to a boom in demand for developers. New software was being created to replace supposedly outdated software that couldn’t be fixed (narrator: “It could”) and salaries were rocketing. I took advantage of that boom. I also knew it wouldn’t last. And it didn’t. My day rate as a PeopleSoft developer went from £200 a day in 1997 to £600 in 2002. It could have been higher. Cisco did an amazing job of raising funds in that era and I remember they kept offering me more and more to go to work for them in the Netherlands. But I didn’t really want to go to work there. I never really chased the money, so that’s about where I peaked. But I remember people with the right skills, experience and self confidence were on as much as £1k a day. That’s getting towards £2k a day at today’s prices. Some skills seen as super hard and rare could command double that. Most people didn’t, of course, make nearly that much, and some people preferred a job with reasonable hours and close to their families – a very valid and decent decision. But I was single with no ties.

There are a lot more developers around today – good incomes have brought many people into the trade. I meet people who called me a nerd in the eighties and now they’re working in IT. It’s a bit weird.

Today’s situation

Now it’s a bit weird. Rates still aren’t at the dotcom level, once adjusted for inflation, but they’re close. You can do very well in tech. But in my little firm we pay typically around £40k for a developer, plus various benefits, kit, resources etc, meaning you’d need to make around £70k as a freelancer to equal it. At least the way I calculate things and always did. I nearly swapped my £600 a day for £60k a year and kind of regret not doing that.

But why have the rates risen? Well, there are a few hot areas, and they can be summarised as AI, analytics, mass market apps, and blockchain. I’ll discuss each briefly:

AI

This is a hot one – the idea we can replace rooms full of people doing dull and not very high value work (from the perspective of the company) such as service desks with AI bots is very attractive. It won’t work though. Most “supposedly AI” bots are just following decision trees and the only bit of AI is in parsing the meaning out of a sentence in a very tightly defined context. AI is useful today for categorisation problems – e.g. looking at a picture and deciding “this is a cat” or “this is a threatening comment”. It’s not brilliant at the job, but I like that an AI can work out which pictures are of my Mum, for example, even if it misses about a third of them… it still makes my life easier. A bit. But what an AI can’t do is right a decent blog post. Sorry, it can’t. They’re awful at it. There’s loads of AI generated content out there and it feels obviously fake. The main job of these AI generated blog posts is to trick other AIs (Google, Bing etc) into categorising a website as useful. And because AI’s make toddlers look worldly wise, they can be easily fooled… and that means you can’t trust them with anything of real importance. Like your business decisions.

But, it’s a hot keyword, and naive venture capitalists like the idea. So in comes the money.

Analytics

Tracking and stalking customers across the internet is very attractive for advertisers believing that doing so makes them seem more interesting to consumers. I’m not convinced. People often find it creepy. They feel like they’re constantly stalked. They visit the website of, say, a printer supplier and they receive ads for a month for printers… but not only for that supplier, but for other printers because the tracking provider is cheerfully using your data as a supplier against you and selling that information to your rivals! I think advertisers are starting to cotton on, but are unsure of what to do… but I know there’s a lot more direct selling of adverts between publishers and advertisers than there used to be.

But, the siren call of analytics is strong, and people love a nice chart on which to justify a decision, so the more nice charts your system can create, the more people will pay to use it and try to gain an advantage over competitors. And advertising is huge, so in pumps the money. For now.

Mass market apps

Can you build the next Facebook, Instagram, or Slack? What’s the potential for an app that lets people read books from any publisher for a fixed monthly fee? How about an app that revolutionises food delivery? Interestingly, some apps are about replacing old and inefficient intermediaries and putting new ones in place. Uber is a nice way of hiring a minicab with flexible pricing that rewards drivers for being available at the right time. They don’t disintermediate, however. The customer is both the driver and the passenger. The new intermediary takes their share.

If you can replace old intermediaries you can make a lot of money. Imagine taking 0.5% of every single financial transaction, like Visa do? That’s a lot of money. Then you have intermediaries between the card firms, providers, and networks, such as Stripe… and then there are those replacing old ones, like Wise, for money transfers across borders.

What other things can be improved? Well, literally anything.

But most attempts to build these apps and the supporting infrastructure are doomed to never turn a profit.

Blockchain

Blockchain is a really interesting concept for a public ledger, using an interesting concept called proof of work to make it hard for any one person to try to dominate the network and win the consensus mechanism on new transactions. There are theoretical ideas out there to improve on this, but at the moment they remain just that and haven’t been proven.

And it’s a scam. Pure and simple. But it’s a hot topic. Bitcoin, Ethereum, Dogecoin and many others are actively speculated upon, as well as being used for the exchange of value – often in a hope to evade regulators. It appeals to the natural rebels amongst us because it’s outside of government control… and given that governments aren’t always a force for good, I get that.

Problem is, Blockchain breaks the rules of good software development… if you look at the big O notation for software, it has to follow certain rules or it will fail at some point and need to be re-engineered. Big O matters. I don’t have academic access to papers, and the internet is full of vested interests pretending that Blockchain scales just fine. I used to see the same in WordPress land, where people said the software scaled fine… but it doesn’t. In WordPress we get scale by putting a layer between WordPress and the internet to balance things out – the work the software itself does goes up in line with the number of people talking to WordPress. We can define that as O(n) so long as you know what you’re doing – that’s OK. We can live with that. But the consensus mechanism required for multi node agreement of transactions as required to track transactions will, by its nature, follow a curve that is likely to be somewhat greater than O(n^2) (each node does O(n) work in a linear fashion but the total work done on the network as each node is added therefore grows as O(n^2) plus a bit for network latency and overheads. Yet bitcoin transaction cost isn’t following that curve in spite of huge interest because, I reckon, most Bitcoin trades aren’t real.

Yes, that’s right. And what does that mean? It’s because wideboys, crooks and the overly-optimistic are involved. Given it is, by design, a pyramid scheme, it will have to fail at some point. But people are motivated to hide that, so there are Bitcoin tracker schemes, rather like gold purchase schemes, that never hold the asset in question. They will pump and pump values as hard as you like. And as long as there are new people coming in, like our receptionists wishes to, all is good.

And there are enormous amounts of money to be made. As in a goldrush, the people making real money are the shovel makers and traders. And they need developers. So for as long as there’s money to be made, coked up wide boys will be gurning their way through stressful meetings, fidgeting and anxious to cash in before it crashes out. You can earn a lot there. For a while.

OK, so thanks for the very long essay. What does it mean then?

Well, it means developers are really expensive right now. Small firms that do actual useful work and aren’t highly capitalised (like mine) can’t grow because we can’t suddenly charge our customers double for the work so that we can compete against these booms. It’s as if a very rich person has moved into your town and hired all the builders possible to create a huge mansion. They even approached builders working for firms and offered them double to come build that mansion. Soon builders are all swanning around town in Teslas and feeling pleased with themselves for being so cunning as to be in the building industry.

Same in software. Locally there’s a Tesla with a crypto referencing private number plate and a young, bearded and muscular techbro driving it. Fine, I’m not going to judge. He’s happy and making good money.

But if builders are all hired by the rich, the rest of us get priced out. Same in software. Small firms are going to find they can’t afford websites unless they just use some cheap web builder platform – it’ll give a less good solution, but it’ll do the job. Ish. And the firms that can afford will do that bit better. And better. And the gap will grow.

At my firm I’ve had to raise salaries, but we still struggle to clear a profit with the raised salaries. I’m fiscally conservative, so we’ve always had decent cash reserves. This lets us ride out the storm. From 1997 to 2002 dev rates went crazy. By 2005 they were back to normal again. We as a firm can’t handle eight years of this. But it’s not quite the same as back then – you can now hire developers globally and have them work remotely, if you really wish to, which can save some money and also help those countries out with extra foreign revenue. I, however, really like quality and good communications and I find that a geographically tight team works the best. It also makes it easier to hire new people into the trade. So, for now, I’m sitting tight. I won’t seek venture capital, or borrow. And if the worst comes to the worst, we’ll add AI to something that does basic statistical analysis, and blockchain to something with two computers in the network and hope someone out there fancies throwing us some money so we join the party. In the meantime, however, there’s still a healthy living to be made as a business doing useful things and avoiding the hot trends. I never set out to be rich, merely secure – I’ll ignore the rich mansions and do my own thing, creating good code for good people.

n.b. about the above – the above isn’t a paper. It’s a set of opinions designed to inform and illuminate about what’s happened. It relies on anecdotes. Don’t take it too seriously and don’t use it as the basis for what you want to do with software and investing in software. Or crypto. Do your own thing with the information you gather from multiple sources. Also remember that a lot of people say misleading things because it’s in their interests to do so, and that you shouldn’t trust a random blog or news source on the internet. Mine included.

Blog “Reboot”

Hello – here’s the refreshed blog. I’ve decided to revert to a more typical blog format, after many months of soul searching on the issue. I previously had a layout based on a framework we used at interconnect/it for a couple of clients

But not only have I opted to switch to a blog layout, I’ve decided to use an off-the-shelf theme.  I’m now using Khoi Vinh‘s Basic Maths WordPress theme.

Why?

Well, it’s a lovely theme, for starters.  The typography is pretty good.  The archives page is brilliant (check it out) and should be the standard bearer for all themes archive pages.

But the real question for many, I suspect, is why I’m not using an interconnect/it designed theme.  Well, for starters, interconnect/it hasn’t produced an off-the-shelf theme in years.  It’s just not our business.  So rather than use a product of ours, we’d have to spend good and valuable time on creating a new theme.  And, well, why would we want to do that?

Lots of reasons, actually.  I could have a theme coded at the office that really shows off what we can do.  But the problem with that is that there’s not much need.  My blog is not an important one.  It isn’t about WordPress (most WP related content will be on our company site, not my personal one) and it just doesn’t get much traffic.

I run a business.  Its purpose is to make money, employ five people, and, with a bit of luck, turn a reasonable profit.  Its job is not to service my ego or make me look good.  A really good theme costs the equivalent of around £10k-£20k of chargeable time to design, code, test and implement.

Given that we’re turning work away, I thought “why bother?”  And decided to go shopping for something.

So What’s It Like?

It’s actually quite weird using somebody else’s theme.  I actually tried a few out and here are the things I learned that will hold us in good stead.

Themes don’t do enough to make life easy.

No really, they don’t.  One of interconnect/it’s biggest challenges is making sure that WP is as easy to use for clients as possible.  This means following standards, but it also means using some little tricks that help out – for example, registering and setting plenty of different image sizes, and setting/over-ruling whatever the media settings say.

Migrating WP content really sucks.

There’s a fundamental flaw with the default WP export/import.  If you have inline images, although the importer has the ability to download and attach the image in your new site it won’t change the links.  And if you do a search and replace, and your image sizes have changes, your lost.  Totally – the img tag will point to a file that doesn’t exist.

So what do you do?  Well, usually if I’m moving a site from one server to another, even switching domains, it’s a non-issue.  I have my tools.  But if you’re starting from fresh and working like an end-user would then you have to go through every single damn post in order to fix the images.  Every post with an image in it.  That’ll take a while.

If you’re really geeky, you’ll sort it, but it takes time.  Way too much time.  This kind of stuff needs to be sorted and it’s something we may look into as a contribution to the WP project.

Some Plugins Leave Lots of Crud

The reason for a reboot was that I felt that my site’s DB had been filled up with all sorts of crud.  Lots of plugins create tables, leave options, and so on.  Surplus tables have little impact, but they clutter the place up.  But options, lots and lots of them, do have a minor performance hit, and they add up.

Other plugins leave hooks, don’t deactivate properly and so on.  And over the years, I’d been through an awful lot of plugins.  The site hadn’t been redone since WP 2.0 had been set up on it.  I felt it was getting sluggish.

So… there are beautiful and amazing themes out there, and WP is wonderful, but there are little things that could make life just that bit better.  Better migration tools, a better system of managing images within content and their migration, and a better system for activating themes so that image sizes are better handled.

Is it a lot to ask?  Well, we’ll see what we can do about that!

Thinking Digital University (2011)

So, here I am again at Thinking Digital.  Only this time I’m no longer driving the seemingly doomed Golf TDI I had last year that did one of it’s self destruction tricks en-route.  Consequently I’m not missing out on the workshops here.

In fact, I’m doing better than that – an additional workshop was added for the Monday by Jer Thorp of Wired fame.  A workshop on Processing.  That, I must say, was a wonderful find.  Processing, in case you’ve never heard of it, is a data visualisation tool or sketchbook.  It’s a bit old-school, but this is a good thing, generally, because this has the advantage of being relatively accessible.  In fact it reminded me of the fun early days of BASIC on small computers.

Simply put, you can easily draw things, and you can analyse data with it.  Some was stuff I could do on a Dragon 32 nearly thirty years ago, but with many thousands of times the power – and that means you can do cool stuff in real time.  I recommend you look up some of the online Processing materials.  You can even try it out without installing anything by using my colleague Robert O’Rourke‘s website, hascanvas.com

During Nancy Duarte’s Workshop

That Resonates With Me!

Then on day two it was a half day ‘off’ which, for me, meant a series of telephone calls with clients while I ensure that work continues as it should.  The afternoon, however, brought along Nancy Duarte‘s “That Resonates With Me!” workshop.

Funnily enough, her resonate analogy was the one bit that didn’t work for me.  She used the peculiar patterns of salt as it’s vibrated on a plate as a way of showing how different people can resonate with your message in different ways.  It’s interesting, but I feel that people don’t work that way.  People can, however, be like salt – you know, small, hard, square and bad for your health.  So perhaps she had a point.

BUT – I’m picking.  Because truth be told it was a fascinating workshop that helped me to see through the clutter of my presentations and to find ways to understand my audience and find ways to connect with them.  The simple exercise she gave will help me improve my presentations – of that I’m sure.  I just have to make sure I put them into practice.

The Rest

The rest of the conference is more classically organised, with the usual talks, networking and information overload.  In the evenings there’ll be the usual entertainment.  Already I’ve been better at avoiding alcohol than last year – I’m remarkably sober tonight.  This is a Good Thing.

Highlights, I suspect, will be Jer’s talk (always visually amazing – check out his Vimeo feed) but the rest I’ll have to report on later.

New Spectacu.la Discussion Updates

My colleague James has been extending the threaded comments plugin that I use(d)* on this site.  It’s available from http://svn.wp-plugins.org/spectacula-threaded-comments/trunk for those of you with SVN clients, and for download and easy installation from WordPress.org and Spectacu.la within a few days.

If you wish to test it out, feel free to comment here…

It adds a quote button for content (try selecting some text here and see what happens!), an extending comment box, so if you like people to write long comments they’ll find it easier, and a quote button for comments.  These features all enhance the WordPress commenting engine and make it easier for your community to engage with you.

A proper release after final testing and updates is due in a few days.

* Still a good plugin that we use extensively – but on this site we designed something custom. Better that way.

WordPress Performance, Make it 3x Quicker!

I’d started to notice that my site could often be slow to load – other sites on the same server weren’t suffering the same way, so I wanted to document a simple way in which one can identify performance issues on the site. This is one of them.

A little while ago I reported that my site, since some WordPress upgrades, had started to slow down. I’d wondered whether it was WP becoming increasingly bloated, or some other problem.

Well, it took me a while to get back to the issue (babies and a booming business don’t help!) it’s continued to get worse and worse, until a recent change has improved things… but only marginally, as shown by the Pingdom chart below:

Not looking good…

This is dreadful, really – daily average of 4,000ms responses just aren’t acceptable where, two years ago, I was getting 800ms.

So, now the process starts.  The recent small improvement came after installing our Spectacu.la Advanced Search Plugin, which runs a regular database optimisation to help keep things nippy, but it was still dreadful.

Is it Pluginitis?

My first suspicion is always that of plugins (and sometimes themes, if they’re complex).  In our office we have a term called ‘pluginitis’ which refers to the problem of a site having too many plugins installed, many of which are poorly written.  I hate to say it, but when clients call to ask for a plugin to be installed that we’ve never tested we go through it and, 90% of the time, discover serious performance or security flaws that will cause long-term issues.

And this site here is old – I’ve been running a WP install for four and a half years with nothing more than upgrades and, like an old PC that’s been upgraded too many times, that causes issues with old drivers and code.  Same can apply to WordPress.  So let’s see what we can do to improve things.

First stage is to disable as many plugins as possible so as to isolate the issue.  I’m using a division based approach – ie, I’m going to disable half of my plugins to see what happens.  If I get full performance back, then the problem lies in that half.  I can then reactivate half the plugins and see what happens.  If the performance is still good, the problem is in the other half.  I think you can see where I’m going here.

I’m also going to go for plugins that aren’t written by us. Not because I’m biased (ok, maybe a little) but because I know all of ours are carefully tested for performance – many are run on major sites such as the Telegraphs blogs site.  Speed is of the essence.

I’m also going to skip plugins like Akismet, because anything that’s essentially ‘core’ is usually going to be reasonably performant – at least on a small site like this one.

It’s worth noting that I could easily delve into SQL statements and code efficiency – but that’s only interesting to developers – if you’re simply a WordPress user, performance is interesting but what you can do to find problems is somewhat more limiting.

Plugins being disabled:

Add to feed – a simple plugin, but sometimes simple plugins miss simple tricks.

Headspace2 – I have my suspicions about this plugin as it’s massive. Could be fine, may not be.  Only way definite way to know – measure it.

Search Meter – a nice plugin to see what people are searching for, but is it adding load somewhere?

Social Bookmarks – it shouldn’t cause issues, but you can never be sure.

wp-typography – I love what it does for the typography on the site, but it’s also running a lot of javascript.

First results:

I do use YSlow to test the site, but one of the problems is that it’s hard to get a large enough series of data to be statistically relevant.  It’s good for seeing the extra load (and why I knew the amount of javascript was an issue) but for longer term analysis it’s flawed.

So, we go back to Pingdom and look at the one day chart.  As I type this it’s now an hour since disabling the plugins above – so let’s see what’s happened:

A dramatic improvement!

As you can see, in this afternoon alone there’s been a dramatic improvement – from around 2500ms per visit to 1230ms per visit.  In one single step I’ve halved the load time of the page.

What we don’t know so far is whether that’s because the page got smaller to load or whether it’s down to a reduction in database load – but that’s really for another article.  What this is all about is trying to document how I’m improving the responsiveness of the site in a way which relatively non-technical folk can follow.

What I’ll do in the next feature is to turn off some more plugins to measure the impact they had.  I’ll also be interested to see if the spikeyness of the response times has varied much – are they caused by simple server load, or is there something else at play?

I will then start to switch plugins on again in a structured way in order to measure which was causing the heaviest loads on the site.

Keep watching!

Interconnect IT

Ok, so you’ll see a new category now under the Asides category. That of ‘The Company.’

And what’s it going to be about? Well, it’s going to cover the tribulations, stresses and joys of building up our web design company, Interconnect IT. The updates won’t necessarily be regular, lucid or sane, but they might be interesting. I won’t even be putting them in the highlight’s category so they show as headlines – you either have to come looking, or you need to subscribe to the RSS feed to this site, where the posts will always show up.

Realistic Pricing

One of the hardest things in business is to come up with a price structure that works. You can easily underprice yourself, convinced that what you do is actually quite easy, and end up trading like mad without actually ever making any money. I had an interesting case this week, where a chap running a fitted furniture company in Liverpool was interested in a website. He’d seen some nice sites from rival firms. Unfortunately he didn’t have an internet connection, so we couldn’t review them together, and the 3G broadband dongle for my laptop is still to arrive. Anyway, we discussed why Interconnect IT is such a great development company, and why he should consider us for a site.

And then there was a risk of it all going downhill. When asked what sort of budget he’d thought of, he came up with the figure of £250. Given that no web design company in the UK can charge less than £45 an hour, he was obviously believing that a site built to professional standards would take about 5.5 hours. Let’s break that down:

Initial meeting – 45 minutes.
Design and layout typography for a simple header and logo – 30 minutes.
Colour pre-defined template to match branding and export for css – 30 minutes.
Create new client directories, copy over notes and so on, make copies of client code – 30 minutes.
Obtain images, with permission for use and prepare them for the six pages required – 1 hour or more.
Purchase domain (finding a suitable name, and get approval) – 1 to 2 hours.
Set up server for the domain, e-mail addresses, security and so on – 1 hour.
Upload server side software (all our sites are dynamic), activate any plugins and so on – 30 minutes.
Option setting on the software – 15 minutes.
Insert supplied text, images and so on, and lay them out smartly – 1 hour.
Test the site on three Windows, two Linux and two Macintosh browsers – 1 hour.
Invoicing, chasing the invoice, banking it and so on – 1 hour.

Now this is for a very very basic site where we’ve done a minimal amount of custom work. As soon as demands grow, so does the time. And we’re also ignoring the development cost of what we’ve done in the past, yet we’re sitting on at least nine hours work for something really quite simple. That gives us £27 or so per hour. And we haven’t even talked about the cost of our server, office, equipment and so on.

We’ve worked it out, and we could actually do about fifteen of these sites per month. Our costs, if we were all earning minimum wage, are about £4000 a month. So we’d lose £250 a month, whilst earning minimum wage.

So what this chap was basically saying to me was that he valued our services at roughly the same level as shelf stackers in a supermarket.

But here’s one thing I’ll admit… it’s not his job to work out the value of what we do. It’s ours. And it’s our job to convince clients that what we do is both highly beneficial to their businesses, and very difficult to do well.