How boring or novel should your technology choices be

Let's talk about technology choices today! It's early weekend here and I just read a nice article on HN titled: "How boring should your team's codebases be". It made me have some thoughts of my own, in regards to what you should strive for:

The author summarizes up what they write about in the first few paragraphs pretty nicely, actually:

Recently I was talking to friend about getting new people up and running in software teams. He mentioned some problems a few new starters had had with an unfamiliar library they’d chosen in one project.

This got me thinking about a pair of blog posts I’d read a number of years ago about novelty budgets and choosing boring technology. These blog posts both use different language to talk about similar concepts: a novelty budget and innovation tokens. The core idea is that you should limit the amount of “non-standard” solutions you use on a project.

To me balancing this level of novelty is very important for a team. Whenever you bring someone new into the team they have to get up to speed on all your existing code and technology choices. The more unusual these choices are the steeper (and/or longer) this learning curve will be.

In no uncertain terms, you need to find the right amount of novelty in your projects.

Too little novelty and you'll be sending PHP code through SFTP to your shared hosting, which doesn't scale that well and will usually severely limit what you can do. I've actually helped people with sites like that and some of those setups get pretty contrived and sooner or later, one bad site or page slows everything else down as well, but migrating away to something else is too hard. Similarly, you might use jQuery in 2022 and wonder why the experience isn't as good as it might be. Or maybe, you need a week or two to set up a new application instance because of all the configuration files and even then there are configuration issues.

Too much novelty and you'll find that your Kubernetes microservice setup will be impossible to reproduce and run locally for debugging, onboarding will be pretty hard because it might turn out that you don't have a large and good enough DevOps/DevEx team to ensure that the developer experience (DX) is actually smooth and you're just generally going to be fighting an uphill battle in an attempt to pad your resume or chase after trends. When things break, ideally it should be clear what happened and how to fix it, instead of prod being down for days.

There are other things to consider, of course, like how easy it is to hire for certain technologies, or whether going for a too boring of a setup will make your skills deteriorate and not keep up with the times, potentially making your kind of unhireable outside of legacy projects, but today I'd like to offer a few of my own examples of the amount of novelty that I find to be "just right".

Applications

When it comes to applications and how they're architected/structured, personally, I'd say that something like 12 Factor apps are a good and supposedly new approach (using mechanisms and approaches that have been around for a while) to let you look at software written in different languages pretty much the same from the outside.

For example, you use environment variables for configuration, write logs to STDOUT, don't cripple your own horizontal scalability by not reaching for local storage (e.g. when S3 might be better suited) or local application memory (e.g. when Redis might be a good idea). It's nice to have those sorts of suggestions in one place and all of the sudden you escape XML or Tomcat setup hell, and can look at Python apps and Java apps similarly from the outside.

Have a look at: 12 Factor Apps

Runtimes and containers

Similarly, adopting containers has been a good solution, both because it allows achieving what people historically didn't bother with when having the opportunity of using systemd, but also because all of the sudden your applications are like phone apps - that can be launched in a consistent format on any server that you need.

And you get health checks, resource limits, automatic restarts, bind mounts, port mapping and internal DNS, all of which you will never build in environments where the DevOps knowledge or resources (time) are not there. While you can have any or all of those with other solutions (systemd slices are particularly underused out there), getting all of that out of the box and more importantly, having those be easy is invaluable.

Note: Kubernetes might be too complex for some setups, as HN loves to point you, something like Nomad or even Docker Swarm also still exists and works, actually it's just a small step up from Docker Compose, the pinnacle of simplicity.

Have a look at: Docker Swarm, Hashicorp Nomad and K3s

Infrastructure and configuration

Speaking of which, infrastructure as code is great!

Using something like Ansible is definitely a novel thing to do at first, but cutting off my team's write access to the servers and making them use GitOps with code review for the configuration changes has been a solid idea. No more wondering why some random configuration exists, or why it was changed N years ago, now you can just look at Git.

No more fat fingering bad changes, and even if you did something like there's also code review. No more risks like Knight Capital of partial deploys and if something like that were to happen, you'd get a CI notification about what's wrong. Now you can also link your pull/merge request with Redmine, Jira, GitHub/GitLab Issues or whatever else you use for change management. Furthermore, if you can deliver Ansible playbooks, no more trying to communicate what needs to be done through 50 page instruction manuals.

Just describe what you need on the server and let those hundreds of actions execute every morning (or after every commit/merge) automatically, ensuring a mostly consistent state - and way more lazily than learning Nix/Guix, which are conceptually superior, but still haven't gotten a sufficient adoption within the industry.

Have a look at: Ansible

Networking

Furthermore, adopting the "ingress pattern" where all of your apps are in some internal overlay network, but talk to the outside world through instances of Apache/Nginx/Caddy/Traefik is brilliant!

No more wondering about how to set up SSL certificates in each of the different application runtimes or even framework versions. No more worrying about setting up rate limits for each application individually, no more worrying about context paths for how things are deployed - you can configure all of that in your web server, even if you don't use a Kubernetes Ingress controller.

The added benefit is that now you're actually more in control of what gets publicly exposed and how. This might even eliminate the occasional vulnerability in your application web server implementation, were it to be available directly. Seeing all of your networking as a set of nginx.conf or similar files is really nice and liberating, especially when each service has a name, like fluttershy_document_submission_api instead of using IP addresses everywhere, like 39.78.120.300.

It's great when combined with the aforementioned Docker Swarm or another solution that gives you overlay networks, ideally even letting you encrypt all of the inter-node communication, which remains private and secure thanks to this, without forcing you to have an SSL cert for each of your service yourself inside of your infrastructure.

And when you can use Let's Encrypt for automatic certificate renewals, you're golden! Of course, if needed, you can still easily run your own CA for internal stuff, or something like client side certificates (either these or a private network are good options for admin dashboards and such).

Have a look at: Apache, my blog post about why use it and also my blog post about how to run your own CA, if necessary

Front end development

Forget something like jQuery from the old days, especially when you'd integrate with numerous low quality plugins that wouldn't even work that well half of the time. Note: while I'm mostly serious here, there are a few cases where it actually makes sense. Don't take me too seriously here.

Just use something like Vue with PrimeVue/PrimeFlex or any other premade component library, with Pinia for state management. You avoid the trouble of using React with Redux (though React Query is nice) or the complexity of Angular, while still getting the benefits of writing small, mostly self-contained application components. No more thousand line JavaScript controllers, no more messing around with global state, or god forbid using something like AngularJS. And with JSX, it actually ends up feeling more convenient to write front end code, in addition to Vue getting hooks right, better than React IMO.

I've seen projects where people got for TypeScript and struggle because only Angular gets the integration with it well enough (but is complex on its own), I've seen projects where people waste lots of time coming up with multi-select or dropdown components, instead of solving business problems. I've seen TailwindCSS seem appealing to people, but them getting slowed down because of the learning curve and still ending up writing primarily CSS/SCSS, just with more mixins, thus missing the point of something like it.

In most cases, you'll want something that gives you the building blocks to do what you need, whatever they might be. There are also web components, like Shoelace, which seem promising, but aren't popular enough to give you a carefree experience with most of the questions you might have also having easily findable answers online.

Have a look at: Vue, Pinia, PrimeVue and PrimeFlex

Back end development

But the actual applications? Most of the time, they should be mostly boring. Something like REST for talking among them, something like JWT/OAuth2 or OIDC for authentication (ideally managed by a turnkey solution, like Keycloak). If there's ever been anything that comes close to an assembly line style of programming, where you have consistent and similar components with only differing business requirements behind them, this should be it.

Log a lot, expect to handle errors, write your code in a way that can have unit tests written to it. Have your app do all of the database migrations that are necessary and use real databases for integration testing (containers that can be removed later, since mocking is a pain and you might end up testing your mock code instead of the implementation), also look into the ability to seed the database with whatever amount of realistic data that you need. If you can't do that, you're not in control of your database model that the app uses, it's in control of you.

So, how about the technologies? I'd say that most of the common languages out there are good picks, since they have established ecosystems and popular frameworks. In web development in particular, I'd urge you to mostly go for the popular languages, so perhaps something out of the TIOBE index and less Haskell, Lisp or other choices that are more out there. When there's something like Go that seems viable, consider it on the basis on how similar it is to what you or the people you can hire already know and how quickly you can get started.

Using Java? Just go for Spring Boot or Dropwizard; something like Quarkus or Vert.X are nice, but not quite ready yet.

Note: I hate Spring Boot with a burning passion because of how much "magic" there is, it's like its IoC is made specifically to make your life harder. But it's hard to argue with it making you more productive, when it actually works. Thus, putting my ego aside on this one and suggesting it as a viable option.

Using Node? Look at Express.js, it does everything you need, with lots of integrations.

Want to use Python? Django or Flask have got you covered.

Are comfortable with PHP? Laravel or something more lightweight like Slim are good. Or maybe Symfony, if you like it.

Are you someone who likes Ruby? The main choice you have is Ruby on Rails.

Similarly, things are also simple for the .NET folk in the audience. Go with ASP.NET.

Every time I've seen someone go for a non-standard solution or actually writing their own framework, it's been an utter dumpster fire. Good luck debugging some uncommented code that has not enough tests when there's a production outage and the few code comments that you might stumble upon are in Lithuanian or something. I'm only half-joking here.

Code that's been written in public and has survived for close to a decade is always going to be better than what you can build in-house, unless you're a large and primarily tech oriented company with a strong development culture, which is almost none of the companies out there. Pick solutions with fewer unknown unknowns, where the issues have already been addressed.

Pick frameworks with code that's tested and has been built by hundreds of individuals, yet is consistent. Frameworks that have security vulnerabilities fixed and have a reasonably slow release cycle, as well as good documentation. Anything else causes you significant risks. If you can, choose the framework that has little magic in it. Micro frameworks are also something that I don't mind, as long as you don't need to write too many features yourself on top of them. Loose or tight coupling is probably a matter of preference.

Databases and storage

So what about databases and data storage? Whatever you do, keep things simple and predictable for as long as you can, primarily scaling them vertically (though stand by servers are welcome). Here, there are mature options to choose from, but please remember about when you should normalize your data and how you should make sure that the database is easily navigable and explorable.

While I talk about technologies a lot, I'd urge you to avoid some practices, like OTLT or EAV patterns. It's really horrible to come across a database, where you have something like link_type (an application enum that corresponds to a table) and entity_id (which corresponds to a table id) for polymorphic links, for example:

SELECT ...
FROM my_table
INNER JOIN other_table
  ON other_table.link_type = 'MY_ENTITY_TYPE'
  AND other_table.entity_id = my_table.id
INNER JOIN another_table
  ON another_table.link_type = 'OTHER_TYPE_EEF_CC'
  AND another_table.entity_id = other_table.id
WHERE ...
ORDER BY ...

Why? Because without knowing about these app level constants, you cannot easily navigate the DB tables. Furthermore, if you can't use foreign keys directly, you'll end up with orphaned data. Go for multiple simpler tables: I'd argue that having 300 tables will be simpler than having 10 tables, if there are clear links between all of them. If not, why even bother with having multiple tables? You might just go with what Reddit did way back and have just two tables in the entire DB. I'd never want to work on a setup like that, though.

Now, onward to technologies!

As a rule of thumb, go for PostgreSQL or MariaDB/MySQL.

If you need data storage, use something S3 compatible, like MinIO.

If you need key-value storage, use Redis.

If you need a message queue, use RabbitMQ.

If you need document storage, either store JSON in your RDBMS or cautiously go for MongoDB.

In each of these spaces, there are one or two established options and using anything outside of those should only be done when you have person-years to throw at every problem, e.g. the expertise and the $$$ to innovate. I've also seen a project, where the choice was made to use an up and coming database solution, the company behind which went defunct.

You don't know the pain of having something running in prod because it hasn't been migrated away from, just to see a 500 HTTP error or something like that for the documentation site. Or not being able to upgrade to a newer OS or container base image, because the software package isn't available in the newer versions, not even talking about vulnerabilities!

Summary

There are always going to be many competent choices out there, many of which won't make too much of an impact. Whether you use Ruby on Rails or Python with Django, you can probably make competent web applications regardless. It usually won't matter too much whether you go for Java or .NET, you can achieve the same in either, outside of a few very niche cases where one or the other will be a clear winner.

That said, there will absolutely be times when a single choice can make or break a project. Picking Kubernetes or Apache Kafka when you don't have the resources to support those will be a bad choice, when a simpler choice would suffice. You might exit from the company and leave a dysfunctional project that has greatly diminished development velocity behind yourself otherwise:

(pictured: your production environment after the last DevOps person who knew how the damned thing works leaves; one day we'll probably need to do daily sacrifices to keep our code working, like in Warhammer 40k)

Explore new options, sure, but not in production projects, that you or someone else will have to maintain. Also, be honest with yourself, about whether you want to make a side project that solves a problem (and possibly makes money for you), or whether you just want to fool around with new technologies. Either is fine, as long as you don't lie to yourself about your motivations. Don't pick a niche solution and try to make a business that will generate income based on Rust if the framework support isn't there yet, or worse yet, you aren't familiar with it and want to learn it.

I remember watching some of Randy's game development videos on YouTube - he said that he's making a game, but he just wanted to write an engine and explore new technologies, so understandably the game was never finished and he scrapped the idea. It was only when he was honest with himself and focused on shipping things, which is when he got things done. Sometimes picking a boring off the shelf solution, like Unity or Godot for game development, is the smart choice. Most of the time, the same logic applies for web development, or other kinds of programming, too.

In most circumstances, just use what has worked for other people well, as long as their circumstances are similar to yours. When all that you need is a boring CRUD app by Friday, don't try to set trends or disrupt the industry. I wish someone had told me that earlier, or other people would listen when I tell that to them and I'm definitely thankful when someone reminds me about that when I get carried away with the possibilities of containers but everything is already working.