This post is mostly a warning to me for the future, but hopefully it'll prevent somebody else from going through the same problem. I've been running a small Kubernetes cluster for a couple of years now, mostly as an experiment and to keep my skills tuned for new tooling. Part …

Recently, I noticed a problem with my Index Lifecycle Management (ILM) not appropriately rotating indexes. The error was not super clear, but I did notice that the existing index had just reached 90 days without closing and that was the first move in the ILM. It was clear that the …

Background Generally speaking, I refresh most of my systems pretty regularly, spurred on by security concerns, general hygeine, a desire to make sure the automation doesn't age out, and certificate expiration. Although I don't need to refersh systems due to certificate expiration, it has historically been the easiest indicator of …

I've already written a bit about using renovate to keep dependencies current using Renovate On Prem in Renovating GitLab Repos. This has been working well. However, there are a couple of twists that I figured I'd document in the event that people run into them. For single-repositories with public dependencies …

Most of the system administration work that I do has been automated using Ansible, as I've mentioned in posts here, including Deploying with GitLab. Now that I've got Renovate in place (Renovating GitLab Repos), I am starting to look at how to expand beyond my existing automations in order to …

I note with some amusement the fact that I wrote on this exact day last year about this same subject (in much more detail). The reason for the new message on this subject is that I'm still cleaning up some of the decisions I made when first using Bacula. The …

I've been an ELK Stack (Elasticsearch, Logstash, Kibana, and Beats) user for quite some time, using exclusively the open source version fo the stack. Generally it's works well and, with some exceptions, supports our mostly-Solaris based environment (using LX zones to host most of the beefier components, and using custom-built …

After 18 months of using Bacula and sending copies of my data to the cloud (in this case, cloud I operate in another location) using an S3-compatible storage mechanism, I noticed I had a lot of data sitting around on my current server for backups. When I set out to …

Back in September of last year, I wrote in Bacula: 6 months on that cloud backups required part.0 in order to be recognized for automatic part retrieval. While this was mostly accurate, the critical file is actually part.1. As such, when referencing my own blog post when trimming …

In June, I mentioned in an article about Docker on SmartOS that we are doing some work with GitLab these days as a replacement for my venerable Gitolite server (and, to an increasing extent Jenkins). Deploying from Pelican I'm likely going to write more on GitLab in the near future …

This spring, there was a some movement on the Illumos/SmartOS front in implementing features to better support running LX zones with Linux variants. Since Docker images (generally) run on Linux underpinnings, support for running Docker images on SmartOS are dependent upon this support working correctly. For those familiar with …

As I've possibly mentioned here before, ClueTrust is using Elasticsearch to perform analysis of log information. Recently, I finally decided to take some our telemetry inforamtion and pull it in to Elasticsearch as a data exploration and statistical tool. Importing structured XML data into Elasticsearch Although there are some filters …

Quite a while back, RS wrote a comprehensive ansible role for handling Let's Encrypt certificate issuance and renewal. We both use this role extensively, which is why it was a significant issue when it suddenly started throwing type errors deep inside of the dnspython library during an nsupdate call in …

Originally this was going to contain a brief Bacula, 6 months on section at the start. Of course, that became much too detailed, so I split them up, however I would encourage you to read it. Restore Testing Backup is the most obvious part of doing backups. Almost everyone's aware …

It's been about six months since I originally wrote Welcome Bacula, describing our transition to Bacula from our previous solution (and a bit of history even before that). If you haven't read it, it might be worth a read. Although not quite 6 months since I wrote the first piece …

We've heard it all before: AWS is expensive, and you need to watch out for the hidden sharp edges of their pricing model. Today I provide a small lesson in that concept. History ClueTrust has run through a number of backup methodologies over the year, originally using Retrospect (when they …

The background Many years ago, all of my systems were pets. I tried to make them easier to manage by standardizing on a single operating system (MacOS X Server at the time) and used management tools that were part of that suite. As time moved forward, Apple decided to concentrate …

I wasn't originally going to write this up on the blog, but considering that we've just finished our transition from our old backup software (BRU, no link) to Bacula community edition and considering that it's World Backup Day, it seemed like it would make sense. As many of you are …

This weekend, Rob and I had been testing the use of hardware keys to secure ssh sessions, especially for back-end console access and certain administrative functions. Since the hardware keys are a special case, and cannot be added to the ssh-agent, we were slinging around a fair number of command …

I've been doing a bunch of maintenance on my two blogs (company and personal) and one purpose has been to track down malformed and mis-mapped URLs on the site. Since both have been through changes in the underlying blog engine a couple of times, there are multiple sets of URLs …