server admin Articles


Monitor fleet aging

Background Generally speaking, I refresh most of my systems pretty regularly, spurred on by security concerns, general hygeine, a desire to make sure the automation doesn't age out, and certificate expiration. Although I don't need to refersh systems due to certificate expiration, it has historically been the easiest indicator of …

Renovating GitLab registries

I've already written a bit about using renovate to keep dependencies current using Renovate On Prem in Renovating GitLab Repos. This has been working well. However, there are a couple of twists that I figured I'd document in the event that people run into them. For single-repositories with public dependencies …

Renovating Ansible

Most of the system administration work that I do has been automated using Ansible, as I've mentioned in posts here, including Deploying with GitLab. Now that I've got Renovate in place (Renovating GitLab Repos), I am starting to look at how to expand beyond my existing automations in order to …

Bacula pruning old storage

I note with some amusement the fact that I wrote on this exact day last year about this same subject (in much more detail). The reason for the new message on this subject is that I'm still cleaning up some of the decisions I made when first using Bacula. The …

Renaming Elasticsearch indexes

I've been an ELK Stack (Elasticsearch, Logstash, Kibana, and Beats) user for quite some time, using exclusively the open source version fo the stack. Generally it's works well and, with some exceptions, supports our mostly-Solaris based environment (using LX zones to host most of the beefier components, and using custom-built …

Bacula pruning

After 18 months of using Bacula and sending copies of my data to the cloud (in this case, cloud I operate in another location) using an S3-compatible storage mechanism, I noticed I had a lot of data sitting around on my current server for backups. When I set out to …

Rclone to the rescue

Back in September of last year, I wrote in Bacula: 6 months on that cloud backups required part.0 in order to be recognized for automatic part retrieval. While this was mostly accurate, the critical file is actually part.1. As such, when referencing my own blog post when trimming …

Deploying with Gitlab

In June, I mentioned in an article about Docker on SmartOS that we are doing some work with GitLab these days as a replacement for my venerable Gitolite server (and, to an increasing extent Jenkins). Deploying from Pelican I'm likely going to write more on GitLab in the near future …

Docker on SmartOS

This spring, there was a some movement on the Illumos/SmartOS front in implementing features to better support running LX zones with Linux variants. Since Docker images (generally) run on Linux underpinnings, support for running Docker images on SmartOS are dependent upon this support working correctly. For those familiar with …

Pivoting Elasticsearch data

As I've possibly mentioned here before, ClueTrust is using Elasticsearch to perform analysis of log information. Recently, I finally decided to take some our telemetry inforamtion and pull it in to Elasticsearch as a data exploration and statistical tool. Importing structured XML data into Elasticsearch Although there are some filters …

Always check your arguments

Quite a while back, RS wrote a comprehensive ansible role for handling Let's Encrypt certificate issuance and renewal. We both use this role extensively, which is why it was a significant issue when it suddenly started throwing type errors deep inside of the dnspython library during an nsupdate call in …

Bacula Restore Testing

Originally this was going to contain a brief Bacula, 6 months on section at the start. Of course, that became much too detailed, so I split them up, however I would encourage you to read it. Restore Testing Backup is the most obvious part of doing backups. Almost everyone's aware …

Bacula 6 months on

It's been about six months since I originally wrote Welcome Bacula, describing our transition to Bacula from our previous solution (and a bit of history even before that). If you haven't read it, it might be worth a read. Although not quite 6 months since I wrote the first piece …

Trapped in the ice

We've heard it all before: AWS is expensive, and you need to watch out for the hidden sharp edges of their pricing model. Today I provide a small lesson in that concept. History ClueTrust has run through a number of backup methodologies over the year, originally using Retrospect (when they …

So much LDAP, so little time

The background Many years ago, all of my systems were pets. I tried to make them easier to manage by standardizing on a single operating system (MacOS X Server at the time) and used management tools that were part of that suite. As time moved forward, Apple decided to concentrate …

Welcome Bacula

I wasn't originally going to write this up on the blog, but considering that we've just finished our transition from our old backup software (BRU, no link) to Bacula community edition and considering that it's World Backup Day, it seemed like it would make sense. As many of you are …

ssh key choices

This weekend, Rob and I had been testing the use of hardware keys to secure ssh sessions, especially for back-end console access and certain administrative functions. Since the hardware keys are a special case, and cannot be added to the ssh-agent, we were slinging around a fair number of command …

Update to nginx_alias_map

I've been doing a bunch of maintenance on my two blogs (company and personal) and one purpose has been to track down malformed and mis-mapped URLs on the site. Since both have been through changes in the underlying blog engine a couple of times, there are multiple sets of URLs …

Client Certs and Intermediate CAs

Why client certificates? RS wrote about Preventing drive-bys with client certs and although we'd discussed this method for some time, I hadn't deployed it yet. However, some recent log-spelunking had led me to determine that I liked the idea of a second layer of protection on some of my sites …

Ansible become: useful and dangerous

OK, now that I have your attention with the catchy title, let me get right into the reason behind this post. Rob has been doing a lot of work lately on a set of roles to provision raspberry pi systems. I'm grateful for the work in this area, because frankly …