Bitnami MariaDB is broken

Here's a short tale about software defaults, drop-in replacements, containers and data loss. You know, there was this one link shortener, YOURLS that to me seemed wonderfully boring and otherwise decent. It supported short links, like Bitly and other services, could be self hosted and also had a really simple architecture - just an admin page for creating links and something like MySQL (or another compatible database, like MariaDB) for data storage. Furthermore, if you clicked on the link above and were greeted by its homepage, you have seen it in action just now.

Now, some people dislike link shorteners because they eventually lead to a lot of links disappearing, should the service itself disappear. In my case I find them to be useful, for the rare cases where links with query parameters might break: for example, they'd be interpreted as instructions for a certain extended Markdown rendering engine, for example my-image.png?resize=1024 works but some-link?article=1234 would break, because it wouldn't be interpreted as the actual link, but rather something for the engine to process. There, links like https://links.kronis.dev/asdf1234 are delightfully simple instead.

Except for me opening the link shortener admin page whilst writing my new blog post and seeing this instead:

This is where I would have proceeded with panicked screaming, were I to have hundreds of links in the shortener, as opposed to a dozen or two, while I was testing how it works (and breaks). So, still in shock and disbelief, I clicked on the only option that was presented to me, to "Install" the link shortener:

That's actually really bad! It installing anew means it assumed that there was no link data stored previously in the database (or some configuration file is wrong) and therefore it probably purged/rewrote everything. This was confirmed by actually looking at the admin UI, which showed me a few of the default links:

This might prompt some folks to get something stronger to drink, but luckily I have backups that actually work!

Backups to the rescue! Or not...?

Here's where BackupPC comes in, a piece of software that I absolutely love for backups (though its configuration was a bit of a mess, admittedly). I just went over to the server in question, looked at the most recent backup and chose to restore the data:

It helpfully let me choose how I want this to be done and then urged me to confirm my choice:

Before doing that, I stopped my applications (which were running in containers, but used bind mounts for their data directory, basically regular directories on the host OS on the server):

And then I proceeded with the backup restore confirmation:

Which then showed me success in the logs (this might have been telling, but I missed the actual details in the logs here):

The database container seemed to start successfully, doing its own thing:

Eventually even the application started as expected and everything became available:

Except for... you know, all of my data!

It was still as if I was initializing a fresh instance, so that made me feel like either the container was wiping some state data upon restart, or something wasn't being persisted correctly. First and foremost, I decided to pull the actual code for the link shortener and do a bit of exploration to figure out the cause for this.

YOURLS is broken! Or is it?

Getting the source code proved to be slow thanks to how SFTP works, I've still no idea why it cannot build a tar.gz archive behind the scenes and send it in one go, but I guess it worked out in the end anyways, so it was good enough:

With the source in my disposal, I set out to figure out why I was being shown the install screen:

So it seemed like it was checking the status or something in the database, which was attached to some class:

The value of which was initialized elsewhere:

Which was actually pulled from a simple SELECT query against the database:

I'm not sure why so many layers of indirection were needed but whatever, I figured out where the problem lies! Something in the database was probably wrong and it's most likely not a problem with YOURLS itself, as far as I could tell.

Then MariaDB is broken!

Luckily, I could also connect to the DB container on the server through the shell and Docker CLI, then enter the MySQL CLI and check the data myself. What I found proved to be the worst case for something like this:

It would appear that the necessary configuration wasn't in the tables... because the tables didn't exist themselves!

That's pretty messed up, eh? This actually made me go back to the backups and explore what's going on there, making me realize what I had overlooked earlier:

The actual folder that was supposed to store the data for my MariaDB instance was empty! One might think that the backup solution had messed up, but nope, it actually works as it should and gets all of the data as instructed (which I checked with plenty of other applications, users, permission setups previously, across many servers). Instead, I looked at the actual container definitions and bind mounts to figure out where the problem was:

And yet, it seemed correct. Previously I had worked with MySQL containers and also MariaDB ones explicitly set their folder names to the same ones as the MySQL ones (which I had double checked), for compatibility reasons:

Actually, thanks to how containers work, I could check everything locally as well:

What I discovered was that the container could start up, but no data was persisted in the /var/lib/mysql directory as expected. That's when I started looking around Docker Hub and realized my problem.

In the end, it was Bitnami that was broken (sort of)

I had Bitnami to thank for blindsiding me here, thanks to a fault of my own. You see, I try to depend on as few container vendors as possible, generally building most of my containers myself, using Ubuntu as base images, so that I can use package managers of the respective OSes for the most part (e.g. installing Python, Java, Node etc.) and only re-host the more complicated images, like databases.

That's where Bitnami comes in - they have lots of great container images of all sorts, including the databases and other complex software that I want to use.

I did make one huge mistake, however - I read the instructions for the official MariaDB image previously and mostly only glanced over Bitnami's own MariaDB image instructions. In my defense, most of those appear to be the same, at least judging by the environment variables.

However, here's the big difference in Bitnami images:

So essentially, this one change was responsible for causing me issues, because the directories for persistence are as follows:

• official MySQL image: /var/lib/mysql
• official MariaDB image: /var/lib/mysql
• Bitnami MariaDB image: /bitnami/mariadb

While MySQL and MariaDB images are almost 1:1 compatible with one another, Bitnami takes a more opinionated stance here and moves some directories around, for consistency with their other images. This isn't bad per se, but you're going to have a hell of a bad day if you miss this detail.

After fixing the directories, I got an error about permissions:

Which was actually a good thing in my case, because it meant that the directory was actually being used and I just needed to fix the folder:

After this, finally some data started appearing in the folder and database data was persisted as expected:

Summary

What I had to do after that, was manually restore all of the links, though since there were a bit less than 20 of them and I mostly only tried out the link shortener with my blog, that was doable with few to no issues:

So what did we learn from all this?

I would say, that the following:

• the fact that your backup mechanism works doesn't mean that your backups actually work
• Docker bind mounts can be risky, because even if the mount is wrong, the container will still work
• Docker containers that are started/stopped instead of being recreated are a risk for not catching cases like this
• software that should be compatible won't always actually be compatible, even if most of the docs seem like that's the case
• giving new software and setups a short test drive with non-essential data is a really good idea, like I did
• Bitnami is evil (just kidding)

Actually I will keep using Bitnami images in the future, but you can be pretty sure that I tested out restarts quite a few times after this. I was pleasantly surprised that for a change it wasn't YOURLS or PHP that was evil, but actually my own configuration, as well as disappointed at the fact that I couldn't detect something like this a bit earlier. But hey, no harm done and I got a blog post out of it.