Gaige's Pageshttps://www.gaige.net/2024-03-20T03:52:00-04:00Vaulting AWS credentials2024-03-20T03:52:00-04:002024-03-20T03:52:00-04:00Gaige B. Paulsentag:www.gaige.net,2024-03-20:/vaulting-aws-credentials.html<p>I've been describing our Hashicorp Vault journey here at ClueTrust
in a number of posts. Chief among the reasons to use Vault is its
ability to generate and rotate credentials with specific systems and
services.</p>
<p>I've written before about
<a href="https://www.gaige.net/vault-local-testing-setup.html">PostgreSQL credential management</a>
using Vault, which has been quite successful. This …</p><p>I've been describing our Hashicorp Vault journey here at ClueTrust
in a number of posts. Chief among the reasons to use Vault is its
ability to generate and rotate credentials with specific systems and
services.</p>
<p>I've written before about
<a href="https://www.gaige.net/vault-local-testing-setup.html">PostgreSQL credential management</a>
using Vault, which has been quite successful. This has allowed for short-term,
tightly-scoped credentials when interacting with our PostgreSQL servers,
meaning that not only are we definitely not storing credentials in code,
but the credentials we are using are only minimally potent if taken out
of the environment.</p>
<p>This weekend, I began using the
<a href="https://developer.hashicorp.com/vault/api-docs/secret/aws">AWS Secrets Engine</a>
with the intention of creating a similar parttern for accessing our AWS
resources.</p>
<p>Although ClueTrust mostly runs its own physical infrastructure, we do have
some systems that we run in other environments, including
geographically-diverse nameservers we run in AWS. As such, we use AWS
credentials to deploy and maintain those (through Ansible, of course).</p>
<p>This weekend's effort was to move from using ansible-vault-stored
secrets (which I would hand rotate every 3-6 months) to using
Hashicorp-Vault-stored secrets which would be created as needed and
would expire quickly.</p>
<h2>STS Credentials</h2>
<p>I have a fondness for minimized blast radius both in scope and time,
which leads to a preference for using AWS STS tokens (see
<a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html">Temporary Security Tokens in AWS</a>).</p>
<p>Using STS in Vault seems a bit of a trade-off. When you create
your AWS Role in Vault, that role must have the ability to create
and maintain STS tokens, but it must also contain all of the entitlements
that you will be granting in that environment. This can be scoped
per AWS Secrets Engine (so you can do multiple accounts each at their
own endpoint), but the intention here is to trust Vault with as much
privilege as all of the needs you have. Initiallly this may feel
risky, as you're concentrating risk in that one set of credentials.
However, if you were to use multiple static AWS roles and store them
in the same Vault, you'd still have the same blast radius scope, but
each of those credentials would likely have a larger temporal blast
radius.</p>
<p>By using the STS credentials, you can scope minimally in time and
specify (by existing AWS IAM groups, policy ARNs, or a specific policy
document). This gives you flexibility from Vault to give tightly-scoped
credentials when these are issued.</p>
<p>In my case, I used <code>federation_token</code> which provides maximium flexibility
to the Vault management. Also available are <code>assumed_role</code> (which uses
STS and assumes a role which the Vault-assigned AWS User is able to
assume) and <code>iam_user</code> for which Vault creates new users for each
request and then deletes those users after a TTL. Vault also supports
Static Roles, which are similar to static credentials in databases.</p>
<h2>AWS Policies and User</h2>
<p>To enable the use of <code>federation_token</code>, your AWS user needs to have permission to use <code>sts:GetFederationToken</code>.</p>
<p>I created a Policy called <code>Federator</code> as such:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"Version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-10-17"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Statement"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"Sid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VisualEditor0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Action"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sts:GetFederationToken"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="s2">"*"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<p>A second policy (<code>Change-self-access-keys</code>) to allow for self-rotation of access keys:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"Version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-10-17"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Statement"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"iam:ListUsers"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"iam:GetAccountPasswordPolicy"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="s2">"*"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"Effect"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Allow"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Action"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"iam:*AccessKey*"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"iam:GetUser"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"Resource"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"arn:aws:iam::*:user/${aws:username}"</span>
<span class="w"> </span><span class="p">]</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<p>and then a user (<code>vault-federation</code>) which was assigned these two
policies and the additional permissions required to do
actual work.</p>
<p>You can assign these directly, although I did so through
groups.</p>
<h2>Establishing AWS secrets in Vault</h2>
<p>The process of configuring Vault for AWS involves:</p>
<ol>
<li>
<p>Enable your new secrets store in Vault:</p>
<div class="codehilite"><pre><span></span><code>vault secrets enable aws
</code></pre></div>
</li>
<li>
<p>Create the IAM Role(s), Policies, and Users that you need in AWS (discussed below)</p>
</li>
<li>
<p>Register your User (for <code>federation_token</code>, you'll need an
actual user; for <code>assumed_role</code> and <code>iam_user</code> you can authenticate
to a Role) with Vault:</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>aws/config/root<span class="w"> </span><span class="nv">access_key</span><span class="o">=</span>YYY<span class="w"> </span><span class="se">\ </span>
<span class="w"> </span><span class="nv">secret_key</span><span class="o">=</span>XXX<span class="w"> </span><span class="nv">region</span><span class="o">=</span>us-east-1<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">sts_endpoint</span><span class="o">=</span>https://sts.us-east-1.amazonaws.com<span class="w"> </span><span class="nv">sts_region</span><span class="o">=</span>us-east-1
</code></pre></div>
<p>(Note: for <code>federation_token</code> users, you'll want to assign an
<code>sts_endpoint</code> and <code>sts_region</code> to enable use across non-default
regions. It's usually best to choose a region that you frequently
operate in)</p>
</li>
<li>
<p>Once you've registered the "root" user (poorly named, but it is what it
is), you should rotate the credentials so that only vault has them. To do this:</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>-force<span class="w"> </span>aws/config/rotate-root
</code></pre></div>
</li>
</ol>
<h2>Creating AWS roles in Vault</h2>
<p>Now the fun begins. Typical of use in Vault, you'll establish
individual "roles" which can be used to retrieve credentials
from AWS an through Vault policies you'll determine which
users can access those roles.</p>
<p>I tend to create these roles through scripts and check them
in to a git repository, so I will use a command like this:</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>aws/roles/<span class="nv">$role</span><span class="w"> </span>-<span class="w"> </span><<span class="w"> </span>aws/roles/<span class="nv">$role</span>.json
</code></pre></div>
<p>in order to load my roles. An example of the JSON for a role I'm using is:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"credential_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"federation_token"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"default_sts_ttl"</span><span class="p">:</span><span class="w"> </span><span class="mi">300</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"iam_groups"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"max_sts_ttl"</span><span class="p">:</span><span class="w"> </span><span class="mi">3600</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"policy_arns"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"arn:aws:iam::aws:policy/AmazonEC2FullAccess"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"policy_document"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span>
<span class="p">}</span>
</code></pre></div>
<p>I could use the <code>iam_groups</code> list to use one or more
groups as the policy,
<code>policy_document</code> to pass a document containing the precise
policy to assign (assuming it's a subset of the policies
the role has), or (as I'm doing here) pass complete ARNs in a list.</p>
<p>All that remains is to make sure this role is accessible from
your users or roles in Vault (left as an exercise to the reader).</p>
<h2>Using aws credentials from Vault</h2>
<p>Use of the credentials is straightforward:</p>
<ol>
<li>Request the credential through CLI or API<div class="codehilite"><pre><span></span><code>%<span class="w"> </span>vault<span class="w"> </span>write<span class="w"> </span>aws/sts/ec2-admin<span class="w"> </span><span class="nv">ttl</span><span class="o">=</span>15m<span class="w"> </span>
Key<span class="w"> </span>Value
---<span class="w"> </span>-----
lease_id<span class="w"> </span>aws/sts/ec2-admin/NNN
lease_duration<span class="w"> </span>14m59s
lease_renewable<span class="w"> </span><span class="nb">false</span>
access_key<span class="w"> </span>XXX
secret_key<span class="w"> </span>YYY
security_token<span class="w"> </span>ZZZ
ttl<span class="w"> </span>14m59s
</code></pre></div>
</li>
<li>Use these credentials for the next 15 minutes as necessary</li>
</ol>
Poetry in Production2023-11-05T08:58:00-05:002023-11-05T08:58:00-05:00Gaige B. Paulsentag:www.gaige.net,2023-11-05:/poetry-in-production.html<p>I regularly use <a href="https://python-poetry.org">poetry</a> in order to isolate development
environments as I'm putting applications together.
I've been happy with it, and there are a number of methods that I've developed
for using poetry in various environments.</p>
<p>For production, there are a number of different mechanisms used
by people in the …</p><p>I regularly use <a href="https://python-poetry.org">poetry</a> in order to isolate development
environments as I'm putting applications together.
I've been happy with it, and there are a number of methods that I've developed
for using poetry in various environments.</p>
<p>For production, there are a number of different mechanisms used
by people in the poetry community:</p>
<ul>
<li>use poetry directly (<code>poetry run application</code>)</li>
<li>install directly in the root environment after exporting
requirements (<code>poetry export --without dev -o requirements.txt</code>)</li>
<li>use an in-tree virtual environment</li>
</ul>
<h2>Decision process</h2>
<p>I've tried all of them, and you can make them all work. However,
after some investigation, I've decided to land right now on the
in-tree virtual environment for ease of use.</p>
<p>I'd recommend against installing directly in the environment
if possible because of the issues that arise if you need to install
more than one virtual environment on a server. Generally, you'd
think this would be unnecessary, as you should be isolating your
servers anyway, and in a minimalist container environment it is likely true.</p>
<p>In our case, since we use slightly heavier-weight containers
(Solaris Zones), I occasionally have other tools
(like background processes that may be working on the same data)
in the same zone. As such, you can still run into conflicts for
dependencies and the virtualenv isolation brings some benefits.</p>
<p>Once you've determined that you're running in a virtual
environment, the question becomes where to put your virtual
environment data. For development, I prefer to leave it in
the default (cache) directories because it's easier for me
to remake those environments en masse when I upgrade the
python enterpreter(s).</p>
<p>For production environments, the rebuild problem isn't an
issue and the execution environments are generally limited.
At this point, it's really a matter of tidiness and standardization.</p>
<h2>Implementation</h2>
<p>For our Solaris zones, they could go anywhere, but I perfer
the ability to nuke and reconsistitute from source quickly and
without having to go hunt down the virtualenv directory.</p>
<p>In the docker environments that I use for some applications, the
problem becomes a bit more accute. Since I want to create the
install environment using the <code>python-dev</code> container and then
deploy it using a runtime container, that means I need to copy
everything over and a standard location is better for this.</p>
<p>As such, my installation process tends to be:</p>
<ol>
<li>
<p>Set up the in-tree virtual environment:</p>
<div class="codehilite"><pre><span></span><code>pip<span class="w"> </span>install<span class="w"> </span>poetry
poetry<span class="w"> </span>config<span class="w"> </span>virtualenvs.in-project<span class="w"> </span><span class="nb">true</span>
</code></pre></div>
</li>
<li>
<p>Capture the execution environment (which should be in <code>.venv</code>):</p>
<div class="codehilite"><pre><span></span><code>poetry<span class="w"> </span>env<span class="w"> </span>info<span class="w"> </span>--path
</code></pre></div>
</li>
<li>
<p>Export the main-only requirements to a file for installation
(skipping hashes in our environment because we have some home-built
packages that we don't gather hashes on yet):</p>
<div class="codehilite"><pre><span></span><code>poetry<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--only<span class="w"> </span>main<span class="w"> </span>--without-hashes<span class="w"> </span>--output<span class="w"> </span>/tmp/requirements.txt
</code></pre></div>
</li>
<li>
<p>Install the requirements in the virtual environment:</p>
<div class="codehilite"><pre><span></span><code>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>/tmp/requirements.txt
</code></pre></div>
</li>
</ol>
<p>I have used these successfully both in Dockerfile (using
multi-stage builds and copying the <code>.venv</code> over) and in
Ansible for deployment in our Solaris zones environments.</p>
<h2>Ansible</h2>
<p>This isn't a complete ansible playbook, but it should give you
an idea of how to construct an effective one:</p>
<div class="codehilite"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Install poetry</span>
<span class="w"> </span><span class="nt">pip</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry</span>
<span class="w"> </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">present</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Set up the in-tree virtual environment</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry config virtualenvs.in-project true</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span>
<span class="w"> </span><span class="nt">chdir</span><span class="p">:</span><span class="w"> </span><span class="s">'{{</span><span class="nv"> </span><span class="s">program_base</span><span class="nv"> </span><span class="s">}}'</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Capture the execution environment</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry env info --path</span>
<span class="w"> </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry_env</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span>
<span class="w"> </span><span class="nt">chdir</span><span class="p">:</span><span class="w"> </span><span class="s">'{{</span><span class="nv"> </span><span class="s">program_base</span><span class="nv"> </span><span class="s">}}'</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Export the main-only requirements to a file for installation</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry export --only main --without-hashes --output /tmp/requirements.txt</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span>
<span class="w"> </span><span class="nt">chdir</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">poetry_env.stdout</span><span class="nv"> </span><span class="s">}}"</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Install the requirements in the virtual environment</span>
<span class="w"> </span><span class="nt">pip</span><span class="p">:</span>
<span class="w"> </span><span class="nt">requirements</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/tmp/requirements.txt</span>
<span class="w"> </span><span class="nt">virtualenv</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">poetry_env.stdout</span><span class="nv"> </span><span class="s">}}"</span>
</code></pre></div>
<p>In this case <code>program_base</code> is the directory where the <code>pyproject.toml</code>
file is located.</p>
<h2>Docker version</h2>
<p>In the Docker version, you'd use a multi-stage build to create the
<code>.venv</code> and then copy it over to the runtime container. Here's a
simplified example:</p>
<div class="codehilite"><pre><span></span><code><span class="k">FROM</span><span class="w"> </span><span class="s">python:3.9</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="s">python-dev</span>
<span class="c"># Install poetry</span>
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>poetry
<span class="c"># Set up the in-tree virtual environment</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
<span class="k">RUN</span><span class="w"> </span>poetry<span class="w"> </span>config<span class="w"> </span>virtualenvs.in-project<span class="w"> </span>true
<span class="c"># Capture the execution environment</span>
<span class="k">RUN</span><span class="w"> </span>poetry<span class="w"> </span>env<span class="w"> </span>info<span class="w"> </span>--path<span class="w"> </span>><span class="w"> </span>/tmp/poetry_env_path
<span class="c"># Export the main-only requirements to a file for installation</span>
<span class="k">RUN</span><span class="w"> </span>poetry<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--only<span class="w"> </span>main<span class="w"> </span>--without-hashes<span class="w"> </span>--output<span class="w"> </span>/tmp/requirements.txt
<span class="c"># Install the requirements in the virtual environment</span>
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>/tmp/requirements.txt
<span class="c"># Runtime container</span>
<span class="k">FROM</span><span class="w"> </span><span class="s">python:3.9</span>
<span class="c"># Copy the virtual environment from the python-dev stage</span>
<span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>python-dev<span class="w"> </span>/app/.venv<span class="w"> </span>/app/.venv
<span class="c"># Set the virtual environment as the default Python environment</span>
<span class="k">ENV</span><span class="w"> </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"/app/.venv/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
<span class="c"># Copy your application code to the container</span>
<span class="k">COPY</span><span class="w"> </span>.<span class="w"> </span>/app
<span class="c"># Set the working directory</span>
<span class="k">WORKDIR</span><span class="w"> </span><span class="s">/app</span>
<span class="c"># Run your application</span>
<span class="k">CMD</span><span class="w"> </span><span class="p">[</span><span class="s2">"/app/.venv/bin/python"</span><span class="p">,</span><span class="w"> </span><span class="s2">"app.py"</span><span class="p">]</span>
</code></pre></div>
<p>This is a simplified example, but it should give you a good start.
If you are installing only modules (for example if your build
steps result in wheels), you will need to make some modifications.</p>
Renovating git tags2023-10-21T18:56:00-04:002023-10-21T18:56:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-21:/renovating-git-tags.html<p>I've been very happy using <a href="https://www.mend.io/renovate/">Renovate</a>
(the free version) for use on my personal projects. I've previously
discussed running it on one of my k8s clusters.</p>
<p>Today, I was trying to deal with a very specific problem: I needed
to track a dependency via git tags, instead of tracking the …</p><p>I've been very happy using <a href="https://www.mend.io/renovate/">Renovate</a>
(the free version) for use on my personal projects. I've previously
discussed running it on one of my k8s clusters.</p>
<p>Today, I was trying to deal with a very specific problem: I needed
to track a dependency via git tags, instead of tracking the head
of the main branch.</p>
<p>Originally, I expected I'd be able to just set the branch in the
<code>.gitmodules</code> file and then it's do "the right thing." Turns out,
not so much.</p>
<p>I tried a number of ways to leverage the default configuration, but
couldn't get that working. So, I decided to take matters into my
own hands and use a custom config.</p>
<p>Since my usual config enables <code>git-submodules</code>, I need to disable it
in for my custom manager to work.</p>
<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="nt">"git-submodules"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"enabled"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"customManagers"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"customType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"regex"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"fileMatch"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"(^|/)\\.gitmodules$"</span><span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"datasourceTemplate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"git-tags"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"matchStrings"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"url = (?<depName>.*?)#(?<currentValue>.*?)\\s"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"versioningTemplate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"semver"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"depTypeTemplate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dependencies"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
</code></pre></div>
<p>For those who haven't dealt with the <code>customManagers</code> before, they're
very powerful. Basically, you can use RegEx to extract data from the
file and describe exactly which <code>datasource</code>, <code>versioning</code> and more
you want applied.</p>
<p>In this case, I'm pulling the <code>url</code> and looking for a <code>#</code> to indicate
the tag. Originally, I used the standard <code>branch</code> mechanism, but for some
reason, the result of the first application of the <code>branch</code> version
resulted in the URL with the <code>#</code> marker.</p>
<p>Previously, I'd used this in some of my docker files, based on the
<code>regexManager</code>, which honestly is basically the same thing. I'm not sure
why there are two ways to do this, nor why <em>both</em> are in the documentation,
but the other place I've done this is:</p>
<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="nt">"regexManagers"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"fileMatch"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"(^|/)Chart\\.yaml$"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"matchStrings"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"#\\s?renovate: image=(?<depName>.*?)\\s?appVersion:\\s?\\\"?(?<currentValue>[\\w+\\.\\-]*)"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"datasourceTemplate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"docker"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
</code></pre></div>
<p>And I also used it in
<a href="https://www.gaige.net/renovating-ansible.html">Renovating Ansible</a> where I used
it to update based on a gitlab tag, but in an explicit manner, not
as part of a git submodule.</p>
<p>The key difference here appears to be the manager name (and differences in
how I ran the tests).</p>
Sonoma Arq warning2023-10-15T15:09:00-04:002023-10-15T15:09:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-15:/sonoma-arq-warning.html<p>After upgrading to Sonoma, I started occasionally (and then
repeatedly) noticing warning messages and errors related to
cloud files in my laptop and desktop machines in the area
that is for iCloud. The specific files aren't important, although
they seem to be related to applications (mostly on the phone) that …</p><p>After upgrading to Sonoma, I started occasionally (and then
repeatedly) noticing warning messages and errors related to
cloud files in my laptop and desktop machines in the area
that is for iCloud. The specific files aren't important, although
they seem to be related to applications (mostly on the phone) that
have transient caches.</p>
<p>However, to stop the myriad warnings/errors, I found a new setting
in the backups for:</p>
<blockquote>
<p>When a dataless ('cloud-only') file is encountered:</p>
</blockquote>
<p>With three settings:</p>
<ul>
<li>Materialize: pull a copy to the system before continuing (Arq
warns this can be time-consuming and data-intensive)</li>
<li>Report an error: this is the default and puts an error in your
logs (and sends an email if you have emails sent on errors)</li>
<li>Ignore: just ignore this file and don't back it up</li>
</ul>
<p>At home, where time and bandwidth aren't as important as a clean backup,
I selected <code>Materialize</code> and on my laptop, I selected <code>Ignore</code>, since
I don't want to slow things down on the road. The most important items to
back up from the road are the ones I'm creating or editing there.</p>
<p>With those changes in place, things seem to be working fine.</p>
Booting Dell servers over SMB2023-10-13T04:29:00-04:002023-10-13T04:29:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-13:/booting-dell-servers-over-smb.html<p>The first time I did this I didn't document it very well, causing the next
time to be more time consuming, so her'es the rundown.</p>
<p>It's not a secret that we use some older Dell hardware as servers in our
datacenter. We've been pretty happy with it since switching away …</p><p>The first time I did this I didn't document it very well, causing the next
time to be more time consuming, so her'es the rundown.</p>
<p>It's not a secret that we use some older Dell hardware as servers in our
datacenter. We've been pretty happy with it since switching away from HP
and one of the reasons is that the iDRAC system seems much more stable,
useful, and featureful than the HP counterpart. (Oh, and HP puts their
firmware updates behind a paywall which is not desirable).</p>
<p>Setting up a server to boot over SMB using an ISO CD/DVD image is relatively
trivial, but does require a bit of preparation. You can also boot over NFS,
but I've found that a bit less relaiable in the past, and honestly NFS is
more painful to administer if you're not actively using it than
<a href="https://www.samba.org">samba</a>.</p>
<h2>Setting up the Samba SMB Server</h2>
<ol>
<li>
<p>Install <code>samba</code> (your OS may vary here)</p>
</li>
<li>
<p>Create an appropriate user (<code>dell</code> in my case)</p>
</li>
<li>
<p>Configure the share. I use a very simple share from /tmp because
it's simple</p>
<div class="codehilite"><pre><span></span><code><span class="k">[tmp]</span>
<span class="w"> </span><span class="na">comment</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">Temporary file source</span>
<span class="w"> </span><span class="na">path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/tmp/</span>
<span class="w"> </span><span class="na">read only</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">yes</span>
<span class="w"> </span><span class="na">public</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">yes</span>
</code></pre></div>
</li>
<li>
<p>Configure the user password (user <code>dell</code> in this case)</p>
<div class="codehilite"><pre><span></span><code>smbpasswd<span class="w"> </span>-a<span class="w"> </span>dell
</code></pre></div>
<p>enter and confirm the password as prompted</p>
</li>
<li>
<p>(Re-)start the smb daemon (example on smartos)</p>
<div class="codehilite"><pre><span></span><code>svcadm<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>samba:smbd
svcadm<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>samba:nmdb
</code></pre></div>
</li>
<li>
<p>Place your files in /tmp for pick up</p>
</li>
</ol>
<h2>Booting on the Dell</h2>
<p><strong>Note:</strong> you need to have an enterprise license to boot like this.</p>
<ol>
<li>
<p>Log in to the iDRAC</p>
</li>
<li>
<p>Navigte to the <strong>Server</strong> view</p>
</li>
<li>
<p>Click on the <strong>Attached Media</strong> tab</p>
</li>
<li>
<p>Fill in the following:</p>
<table>
<thead>
<tr>
<th>title</th>
<th>value</th>
</tr>
</thead>
<tbody>
<tr>
<td>Image File Path</td>
<td><code>//</code><em>IP or domain</em><code>/</code><em>mount point</em><code>/</code><em>file name</em></td>
</tr>
<tr>
<td>Domain name</td>
<td><em>blank</em></td>
</tr>
<tr>
<td>User Name</td>
<td><em>user name</em></td>
</tr>
<tr>
<td>Password</td>
<td><em>user password</em></td>
</tr>
<tr>
<td>Expired or invalid certificate action</td>
<td><code>Ignore</code></td>
</tr>
</tbody>
</table>
</li>
<li>
<p>Click <strong>Connect</strong></p>
<p>After a few seconds, you should get a confirmation of the connection
<em>Connection Status</em> will read <em>Connected</em></p>
</li>
<li>
<p>Go to the <strong>Virtual Console</strong> and make sure your boot sequence checks
for CD/DVDs and then <strong>Reboot</strong></p>
</li>
</ol>
Exploring distroless images2023-10-13T04:29:00-04:002023-10-13T04:29:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-13:/exploring-distroless-images.html<p>Distroless images are all the rage in the container space these
days due to the reduced attack surface. This is great and also
results in much thinner images. But, when an image isn't behaving
it can cause some additional trouble as you try to figure out what
may be missing …</p><p>Distroless images are all the rage in the container space these
days due to the reduced attack surface. This is great and also
results in much thinner images. But, when an image isn't behaving
it can cause some additional trouble as you try to figure out what
may be missing or broken without the ability to access the image.</p>
<ol>
<li>pull the image (if not already present)</li>
<li>run a container (this mounts the image to create the filesystem)</li>
<li>Export the image contents<div class="codehilite"><pre><span></span><code>docker<span class="w"> </span><span class="nb">export</span><span class="w"> </span>hungry_mcnulty<span class="w"> </span>>contents.tar
</code></pre></div>
</li>
</ol>
<p>This will provide the contents of the image and the container, so its
good for debugging</p>
<h2>Image-only solutions</h2>
<p>If you just want to explore the layers and files in the image, you
may find tools like <a href="https://github.com/wagoodman/dive">dive</a>
(available on the Mac through <a href="https://brew.sh">brew</a>) an appealing
solution... well, if you like UI through the terminal.</p>
Flask and vault2023-10-09T09:27:00-04:002023-10-09T09:27:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-09:/flask-and-vault.html<p>When using dynamic database credentials with Flask, we need to make
sure that the flask instance picks up the right credentials, renews
them when necessary, and uses the right roles.</p>
<p>My flask code is pretty embedded with the database changes here, so
pardon the dust, but I think it's relatively …</p><p>When using dynamic database credentials with Flask, we need to make
sure that the flask instance picks up the right credentials, renews
them when necessary, and uses the right roles.</p>
<p>My flask code is pretty embedded with the database changes here, so
pardon the dust, but I think it's relatively easy to follow.</p>
<p>Configuration parameters are either from the config file or they are
taken from environment variables.</p>
<table>
<thead>
<tr>
<th>Parameter</th>
<th>Required</th>
<th>Purpose</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td>VAULT_ROLE</td>
<td>✓</td>
<td>dynamic database role to use</td>
<td>None</td>
</tr>
<tr>
<td>DB_ROLE</td>
<td></td>
<td>role to assume in connection</td>
<td>None</td>
</tr>
<tr>
<td>SQLALCHEMY_DATABASE_URI</td>
<td>✓</td>
<td>URI for datbase</td>
<td><em>no default</em></td>
</tr>
</tbody>
</table>
<p>The application below is named <code>telemetry_ingest</code> and uses
<code>TELEMETRY_INGEST</code> as the prefix for any environment variables
that are used for configuration. This is mostly interesting if you
are going to adapt this code elsewhere, since you need to remember
to pull those out.</p>
<p>Vault use is triggered by the presence of the <code>VAULT_ROLE</code> parameter,
since the vault credentials may or may not be necessary depending
on the environment. If they are present in the config, this code will
push them to the libraries, otherwise they'll come as <code>None</code> and
<code>hvac</code> will use its defaults from the environment or statically.</p>
<p>Authentication data is stored in the <code>auth</code> global in this module and
is initialized when the application starts. The logic to get and renew
the authentication data is in <code>get_vault_credentials()</code>.</p>
<p>Of particular interest is the event handling at the bottom in the
<code>with app.app_context()</code> stanza. This adds event handlers for
<code>do_connect</code> (called before the connection, so we can load the
credentials), <code>checkout</code> (called when a connection is "checked out" to
do something, where we verify the connection), and <code>connect</code>
(where we set the database role if requested). Finally, the standard
configuration is done, registering the blueprint for the actions.</p>
<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Optional</span>
<span class="kn">import</span> <span class="nn">hvac</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">event</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.exc</span> <span class="kn">import</span> <span class="n">DisconnectionError</span>
<span class="n">auth</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">create_app</span><span class="p">(</span><span class="n">test_config</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="c1"># create database</span>
<span class="c1"># create and configure the app</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="s2">"telemetry_ingest.default_settings"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">test_config</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="c1"># If we want to read from py file instead of prefixed variables</span>
<span class="c1"># if os.environ['TELEMETRY_INGEST_SETTINGS']:</span>
<span class="c1"># app.config.from_envvar('TELEMETRY_INGEST_SETTINGS')</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_prefixed_env</span><span class="p">(</span><span class="s2">"TELEMETRY_INGEST"</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># load the test config if passed in</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_mapping</span><span class="p">(</span><span class="n">test_config</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">telemetry_ingest.models</span> <span class="kn">import</span> <span class="n">db</span>
<span class="n">db</span><span class="o">.</span><span class="n">init_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">use_vault</span><span class="p">()</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">if</span> <span class="n">requested_credential</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">def</span> <span class="nf">requested_role</span><span class="p">()</span> <span class="o">-></span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">if</span> <span class="s2">"DB_ROLE"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="k">return</span> <span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s2">"DB_ROLE"</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">requested_credential</span><span class="p">()</span> <span class="o">-></span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">if</span> <span class="s2">"VAULT_ROLE"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="k">return</span> <span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s2">"VAULT_ROLE"</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_vault_credentials</span><span class="p">(</span><span class="n">existing</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">use_vault</span><span class="p">():</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">hvac</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span>
<span class="n">url</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"VAULT_ADDR"</span><span class="p">],</span> <span class="n">token</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s2">"VAULT_TOKEN"</span><span class="p">]</span>
<span class="p">)</span>
<span class="k">assert</span> <span class="n">client</span><span class="o">.</span><span class="n">is_authenticated</span><span class="p">()</span>
<span class="k">if</span> <span class="n">existing</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span>
<span class="n">existing</span><span class="p">[</span><span class="s2">"response"</span><span class="p">][</span><span class="s2">"renewable"</span><span class="p">]</span>
<span class="ow">and</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o"><</span> <span class="n">existing</span><span class="p">[</span><span class="s2">"vault_expire"</span><span class="p">]</span>
<span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">renew_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">sys</span><span class="o">.</span><span class="n">renew_lease</span><span class="p">(</span><span class="n">existing</span><span class="p">[</span><span class="s2">"vault_lease_id"</span><span class="p">])</span>
<span class="n">new_auth</span> <span class="o">=</span> <span class="n">existing</span>
<span class="n">new_auth</span><span class="p">[</span>
<span class="s2">"vault_expire"</span>
<span class="p">]</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span>
<span class="n">seconds</span><span class="o">=</span><span class="n">renew_response</span><span class="p">[</span><span class="s2">"lease_duration"</span><span class="p">]</span>
<span class="p">)</span>
<span class="n">new_auth</span><span class="p">[</span>
<span class="s2">"vault_renew"</span>
<span class="p">]</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span>
<span class="n">seconds</span><span class="o">=</span><span class="n">renew_response</span><span class="p">[</span><span class="s2">"lease_duration"</span><span class="p">]</span> <span class="o">/</span> <span class="mi">2</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">new_auth</span>
<span class="k">except</span> <span class="n">hvac</span><span class="o">.</span><span class="n">v1</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">VaultError</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"lease renewal failed"</span><span class="p">)</span>
<span class="k">pass</span>
<span class="n">read_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">secrets</span><span class="o">.</span><span class="n">database</span><span class="o">.</span><span class="n">generate_credentials</span><span class="p">(</span>
<span class="n">requested_credential</span><span class="p">()</span>
<span class="p">)</span>
<span class="n">app</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"new lease"</span><span class="p">)</span>
<span class="n">new_auth</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"user"</span><span class="p">:</span> <span class="n">read_response</span><span class="p">[</span><span class="s2">"data"</span><span class="p">][</span><span class="s2">"username"</span><span class="p">],</span>
<span class="s2">"password"</span><span class="p">:</span> <span class="n">read_response</span><span class="p">[</span><span class="s2">"data"</span><span class="p">][</span><span class="s2">"password"</span><span class="p">],</span>
<span class="s2">"vault_lease_id"</span><span class="p">:</span> <span class="n">read_response</span><span class="p">[</span><span class="s2">"lease_id"</span><span class="p">],</span>
<span class="s2">"vault_expire"</span><span class="p">:</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">read_response</span><span class="p">[</span><span class="s2">"lease_duration"</span><span class="p">]),</span>
<span class="s2">"vault_renew"</span><span class="p">:</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">read_response</span><span class="p">[</span><span class="s2">"lease_duration"</span><span class="p">]</span> <span class="o">/</span> <span class="mi">2</span><span class="p">),</span>
<span class="s2">"response"</span><span class="p">:</span> <span class="n">read_response</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">new_auth</span>
<span class="k">global</span> <span class="n">auth</span>
<span class="n">auth</span> <span class="o">=</span> <span class="n">get_vault_credentials</span><span class="p">()</span>
<span class="k">with</span> <span class="n">app</span><span class="o">.</span><span class="n">app_context</span><span class="p">():</span>
<span class="c1"># https://docs.sqlalchemy.org/en/20/core/engines.html#custom-dbapi-args</span>
<span class="nd">@event</span><span class="o">.</span><span class="n">listens_for</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">engine</span><span class="p">,</span> <span class="s2">"do_connect"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">provide_credentials</span><span class="p">(</span><span class="n">dialect</span><span class="p">,</span> <span class="n">conn_rec</span><span class="p">,</span> <span class="n">cargs</span><span class="p">,</span> <span class="n">cparams</span><span class="p">):</span>
<span class="k">if</span> <span class="n">use_vault</span><span class="p">():</span>
<span class="k">global</span> <span class="n">auth</span>
<span class="n">cparams</span><span class="p">[</span><span class="s2">"user"</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth</span><span class="p">[</span><span class="s2">"user"</span><span class="p">]</span>
<span class="n">cparams</span><span class="p">[</span><span class="s2">"password"</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth</span><span class="p">[</span><span class="s2">"password"</span><span class="p">]</span>
<span class="nd">@event</span><span class="o">.</span><span class="n">listens_for</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">engine</span><span class="p">,</span> <span class="s2">"checkout"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">validate_checkout</span><span class="p">(</span><span class="n">dbapi_connection</span><span class="p">,</span> <span class="n">connection_record</span><span class="p">,</span> <span class="n">connection_proxy</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">use_vault</span><span class="p">():</span>
<span class="k">return</span>
<span class="k">global</span> <span class="n">auth</span>
<span class="k">if</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">></span> <span class="n">auth</span><span class="p">[</span><span class="s2">"vault_renew"</span><span class="p">]:</span>
<span class="n">app</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"credentials expired"</span><span class="p">)</span>
<span class="n">auth</span> <span class="o">=</span> <span class="n">get_vault_credentials</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">DisconnectionError</span><span class="p">()</span>
<span class="nd">@event</span><span class="o">.</span><span class="n">listens_for</span><span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">engine</span><span class="p">,</span> <span class="s2">"connect"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">set_role_on_connect</span><span class="p">(</span><span class="n">dbapi_connection</span><span class="p">,</span> <span class="n">connection_record</span><span class="p">):</span>
<span class="k">if</span> <span class="n">requested_role</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">with</span> <span class="n">dbapi_connection</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span> <span class="k">as</span> <span class="n">cursor</span><span class="p">:</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">"SET ROLE '"</span> <span class="o">+</span> <span class="n">requested_role</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"'"</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">telemetry_ingest.routes</span> <span class="kn">import</span> <span class="n">telemetry</span><span class="p">,</span> <span class="n">redirection</span>
<span class="n">app</span><span class="o">.</span><span class="n">register_blueprint</span><span class="p">(</span><span class="n">telemetry</span><span class="p">)</span>
<span class="n">app</span><span class="o">.</span><span class="n">register_blueprint</span><span class="p">(</span><span class="n">redirection</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">create_all</span><span class="p">()</span>
<span class="k">return</span> <span class="n">app</span>
</code></pre></div>
Vault local testing setup2023-10-09T08:28:00-04:002023-10-09T08:28:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-09:/vault-local-testing-setup.html<p>When I was confirming the configurations for my vault management of
database credentials, I used a local postgresql and vault server. This
may also be useful for development (especially testing code that
may exercise the vault and database interactions).</p>
<p>This can make it relatively easy to watch all of the …</p><p>When I was confirming the configurations for my vault management of
database credentials, I used a local postgresql and vault server. This
may also be useful for development (especially testing code that
may exercise the vault and database interactions).</p>
<p>This can make it relatively easy to watch all of the pieces and throw
away any side-effects that you don't want in your production or
staging servers.</p>
<ol>
<li>
<p>If you're using vault elsewhere in your system, make sure you clear
out your credentials from any cache while doing this work or you
may accidentally modify a running system.</p>
</li>
<li>
<p>Make sure you have a running postgresql server</p>
</li>
<li>
<p>Start a vault development server: <code>vault server -dev</code>, which will
automatically unseal itself and run in memory, so there's no need
to clean up after yourself (although you'll have to start the whole
process again if you kill the server)</p>
</li>
<li>
<p>Create a new admin user in your local database. I use a separate user
for vault so that I don't run into a problem in break-glass scenarios, but
you can also do that with local user override. I use <code>local-vault</code>,
and for the sake of this example, set the passwrd to `changeme``</p>
</li>
<li>
<p><code>vault secrets enable database</code> to turn on the database secret manager</p>
</li>
<li>
<p>Create the database config:</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>database/config/localpg<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">plugin_name</span><span class="o">=</span>postgresql-database-plugin<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">allowed_roles</span><span class="o">=</span><span class="s2">"*"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">connection_url</span><span class="o">=</span><span class="s2">"postgresql://{{username}}:{{password}}@127.0.0.1:5432/postgres?sslmode=disable"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">username</span><span class="o">=</span><span class="s2">"local-vault"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">password</span><span class="o">=</span><span class="s2">"changeme"</span>
</code></pre></div>
<p>This command will create the record for <code>localpg</code>, setting it up to manage
the <code>local-vault</code> credentials for the local vault</p>
</li>
<li>
<p>Force rotation of the password (this will make it so that you don't know
the root-standin password):</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>-force<span class="w"> </span>database/rotate-root/localpg
</code></pre></div>
</li>
<li>
<p>Now, for each of the users that we want the database to be able to use, we
need to create roles using the parameters we determined previously.</p>
<p>For the ownership user, this is:</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span>write<span class="w"> </span>database/roles/owner-test<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">db_name</span><span class="o">=</span>localpg<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">creation_statements</span><span class="o">=</span><span class="s2">"CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \"owner\" NOINHERIT;"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">revocation_statements</span><span class="o">=</span><span class="s2">"drop user \"{{name}}\";"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">default_ttl</span><span class="o">=</span>60s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">max_ttl</span><span class="o">=</span>60m
</code></pre></div>
<p>The braced variables are replaced by vault, so this will result in a user
with a name, password, and expiration chosen by vault, which has access
to the abilities of the <code>owner</code> role, but only if it is explicitly assumed
using <code>set ROLE owner</code>.</p>
<p><strong>NOTE</strong>: for real use, you want longer TTLs than 60s and 60m, but these
were used here because it is a test environment and we want to verify both
the renewal and expiration of the credentials.</p>
</li>
<li>
<p>Now, you should be able to retrieve the credentials from vault manually</p>
<div class="codehilite"><pre><span></span><code>vault<span class="w"> </span><span class="nb">read</span><span class="w"> </span>database/creds/owner-test
</code></pre></div>
<p>Once you've created the credential, you should be able to access it and
also see the credentials with <code>\du</code> in the <code>psql</code> interface. You should
see expiration times for the credentials you create, and they should
automatically disappear after that time.</p>
</li>
</ol>
Postgres roles and privileges2023-10-09T08:21:00-04:002023-10-09T08:21:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-09:/postgres-roles-and-privileges.html<p>This is part of a multi-part series on using postgres databases, vault,
and a variety of other tools to effect short-lived database credentials
for real use.</p>
<p>As postgres uses user and role interchangably, so will I, although I'll
generally try to use <em>user</em> to refer to a role with login …</p><p>This is part of a multi-part series on using postgres databases, vault,
and a variety of other tools to effect short-lived database credentials
for real use.</p>
<p>As postgres uses user and role interchangably, so will I, although I'll
generally try to use <em>user</em> to refer to a role with login permissions.</p>
<h2>Postgres roles and privileges</h2>
<p>There are some really interesting and powerful capabilities for managing roles
in postgres, and as I've become more familiar with them, I understand why so many
of us are in the habit of granting overly-broad privileges in the database realm.</p>
<p>A few important things to know about roles and privileges:</p>
<ul>
<li>
<p>The owner of a new object is the user/role that created that object.
As such, any default privileges for objects owned by that user must
either be assigned by that user, or assigned on behalf of the user.
Be careful/consistent about the role that creates tables and sequences
so that you prevent problems. Only the owner is allowed to alter tables
and sequences.</p>
</li>
<li>
<p>By default new objects are not provided privileges to existing roles.
In order to ensure that new tables, sequences, etc. are usable by the
roles that we are creating, they must be either granted explicitly each
time a new object is created, or you must have an appropriate default
permission for the <code>public</code> role or an explicit role. <strong>NOTE</strong>: defaults
for new objects are owner-dictated, and thus the
default role must be grated by the owner of the table, or on behalf of
the owner of the table or it won't work as desired.</p>
</li>
<li>
<p>There are rules about <a href="https://www.postgresql.org/docs/current/role-removal.html">dropping roles</a>
that may not be obvious. An owning role cannot be deleted until that owned
object is removed or ownership is given to another role. Additionally any
privileges owned by the role need to be dropped before dropping the role.</p>
</li>
<li>
<p>By default, all users have <code>create</code> privileges in the <code>public</code> schema for
each database. This is <em>not</em> necessary for the creation of temporary tables,
but generally makes it easier for all new users to work with a database.
When designing your access controls, it probably makes sense to revoke all
unnecessary privileges from public
(<code>revoke create on schema public from public</code>).</p>
</li>
<li>
<p>Roles that are not marked <code>noinherit</code> in their definition will be
inherited by any user granted that role unless you explicitly mark
the grant <code>noinherit</code>. This may make sense when you are looking at
privileges that don't grant ownership, but if you are trying to ensure that
you don't have unexpectedly-owned tables, you will want your ownership
roles to have <code>noinherit</code> set or grant them <code>noinherit</code>.</p>
</li>
<li>
<p>It's worth noting that non-owning users with the <code>createrole</code> permission
do not have the ability to directly set their roles to be any arbitrary
role on the system. However, I've found nothing to prevent a user with
<code>createrole</code> permission from granting itself any role on the system except
superuser. Keep that in mind that these granting roles are potent.</p>
</li>
</ul>
<h2>Role design</h2>
<p>Designing an appropriate role structure is complicated and this is not intended
to be one-size-fits-all in the least. However, I hope it serves as a useful
starting-off point. This design expects tob e used with dynamic roles through
vault, and I'll detail that at the end.</p>
<p>The recommended "group" roles are:</p>
<table>
<thead>
<tr>
<th>role</th>
<th>purpose</th>
<th>privileges</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>postgres</code></td>
<td>superuser role (maybe break glass)</td>
<td><code>SUPERUSER</code>, <code>LOGIN</code></td>
</tr>
<tr>
<td><code>vault-admin</code></td>
<td>used by vault to manage dyanmic users</td>
<td><code>CREATEROLE</code>, <code>LOGIN</code></td>
</tr>
<tr>
<td><code>owner</code></td>
<td>create/modify tables</td>
<td><code>ALL on SCHEMA public</code> and <code>ALL</code> on tables and sequences</td>
</tr>
<tr>
<td><code>readwrite</code></td>
<td>read/write to tables</td>
<td><code>SELECT on SCHEMA public</code> and <code>ALL</code> on tables and sequences</td>
</tr>
<tr>
<td><code>readonly</code></td>
<td>read-only on tables</td>
<td><code>SELECT on SCHEMA public</code> and <code>SELECT</code> on tables and <code>SELECT</code> on sequences</td>
</tr>
</tbody>
</table>
<p>NOTE: The <code>readwrite</code> user is granted <code>ALL</code> on tables and sequences.
This may be a bit more than necessary, consider if you want to grant
<code>REFERENCES</code> and <code>TRIGGER</code> tables, since these are frequently unnecssary.
Although the <code>USAGE</code> privilege is separate on sequences, it's not
particularly useful to allow <code>UPDATE</code> without allowing <code>USAGE</code>, as that
won't allow you to use the sequence in a <code>nextval</code> call.</p>
<p>When creating the dynamic roles (assuming use with vault),
there will be three role templates used:</p>
<table>
<thead>
<tr>
<th>role</th>
<th>creation statement</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>owner</code></td>
<td><code>CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \"owner\" NOINHERIT;</code></td>
</tr>
<tr>
<td><code>readwrite</code></td>
<td><code>CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \"readwrite\" INHERIT;</code></td>
</tr>
<tr>
<td><code>readonly</code></td>
<td><code>CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE \"readonly\" INHERIT;</code></td>
</tr>
</tbody>
</table>
<p>The creation statements are all very similar, except for the specifics of
which role is assigned and if they are inherited.</p>
<p>In my case, I'm choosing to use <code>inherit</code> for the non-ownership roles
because it removes the need to <code>set role</code>. However, it's not necessary
and unless you need the (temporary) user name, setting role is probably
preferred.</p>
<p>This structure can be used for any application framework, and with or without
a secrets maanger like vault, but since my example here started as a vault
example, the creation of these items would be sequenced thusly:</p>
<ol>
<li>Create the new database</li>
<li>Create the <code>owner</code> role and grant it appropriate privileges</li>
<li>Create the <code>readwrite</code> role and grant it the appropriate privileges
and defaults in the schema on behalf of <code>owner</code></li>
<li>Create the <code>readwrite</code> role and grant it the appropriate privileges
and defaults in the schema on behalf of <code>owner</code></li>
</ol>
<p>At this point, you have what you need to create the database admin user,
the reader, and the writer. If you're using vault, then use the
creation statements above. If not, you can use similar statements
manually so that if you have multiple users, you can control access
centrally.</p>
Django and vault2023-10-09T08:20:00-04:002023-10-09T08:20:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-09:/django-and-vault.html<p>When using dynamic database credentials with Django, we need to make
sure that the django instance picks up the right credentials, renews
them when necessary, and uses the right roles.</p>
<p>This post includes the background and the necessary code.</p>
<h2>Migration and creation</h2>
<p>Migration and creation provide special problems because of …</p><p>When using dynamic database credentials with Django, we need to make
sure that the django instance picks up the right credentials, renews
them when necessary, and uses the right roles.</p>
<p>This post includes the background and the necessary code.</p>
<h2>Migration and creation</h2>
<p>Migration and creation provide special problems because of modification
of database objects. For this, we either need to assume the role
(as mentioned above) which owns the items, or we need a separate user.</p>
<p>I intend to try using a separate user for migrations in the future, but
for now, I have a single role that the temporary user will assume which
has access to read and write as well as own and maintain
the tables, sequences, etc.</p>
<p>Since the temporary user has the ability to create objects, there
are some <a href="https://www.gaige.net/vaulting-database-credentials.html#ownership-and-roles">ownership issues</a> that will create problems
if those are owned by the temporary user.</p>
<p>I'm using the database option <code>assume_role</code> to assume a permanent
postgresql role after connecting to limit ownership confusion.
Support for assuming roles for a session was
<a href="https://docs.djangoproject.com/en/4.2/ref/databases/#role">added in 4.2</a>
and is effected by adding <code>('OPTIONS': { 'assume_role': 'test-owner'}</code> in the
database definition in your configuration.</p>
<h2>Renewing credentials</h2>
<p>To renew the credentials, we're going to need to wrap the database access
so that new credentials are retrieved both when required.</p>
<p>For this, I took inspiration from the
<a href="https://github.com/aws-samples/aws-secrets-manager-credential-rotation-without-container-restart/blob/main/webapp/app/codecompose/db/backends/secretsmanager/mysql/base.py">AWS Samples for secrets manager rotation</a>
which used nearly the same mechanism, but with Secrets Manager instead of Vault.</p>
<p>Effectively, I created a new database manager in <code>my-app/db/backends/postgresql</code>
using the <code>django.db.backends.postgresql</code> as a base and then in the <code>DATABASES</code>
configuration stanza, I referred to <code>my-app.db.backends.postgresql</code> as the
database <code>ENGINE</code>.</p>
<p>In the code below, <code>DatabaseCredentials</code> are used to store the database
credentials while they are live. The credentials are stored by the
<code>DatabaseWrapper</code> in instance storage, retrieving the credentials
at init time and passing the <code>settings_dict</code> from the original
<code>DATABASES</code> block along so that we can pick up any salient information.</p>
<p>There are a number of vault-related parameters, all prefixed with <code>VAULT_</code>:</p>
<table>
<thead>
<tr>
<th>Parameter</th>
<th>Required</th>
<th>Purpose</th>
<th>Default</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>VAULT_ROLE</code></td>
<td>*</td>
<td>vault dynamic role name</td>
<td>None</td>
</tr>
<tr>
<td><code>VAULT_STATIC_ROLE</code></td>
<td>*</td>
<td>vault static role name</td>
<td>None</td>
</tr>
<tr>
<td><code>VAULT_MOUNT_POINT</code></td>
<td></td>
<td>database secret store mount point</td>
<td><code>database</code></td>
</tr>
<tr>
<td><code>VAULT_ADDR</code></td>
<td></td>
<td>URL for the vault</td>
<td>None</td>
</tr>
<tr>
<td><code>VAULT_TOKEN</code></td>
<td></td>
<td>Token for accessing vault</td>
<td>None</td>
</tr>
</tbody>
</table>
<p><code>*</code>: At least one of <code>VAULT_ROLE</code> and <code>VAULT_STATIC_ROLE</code> must by included.</p>
<p>If either the <code>VAULT_ADDR</code> and <code>VAULT_TOKEN</code> are empty, the <code>hvac</code> library
will provide its defaults, reading first from the environment and then
using static defaults.</p>
<p>The <code>DatabaseWrapper</code> provides an override for <code>get_new_connection</code>,
adding a set of credentials, renewing them if necessary, and
then forwarding along to the underlying wrapper afte rthe credentials
are replaced.</p>
<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">hvac</span>
<span class="kn">from</span> <span class="nn">django.core.exceptions</span> <span class="kn">import</span> <span class="n">ImproperlyConfigured</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">DEFAULT_DB_ALIAS</span>
<span class="kn">from</span> <span class="nn">django.db.backends.postgresql</span> <span class="kn">import</span> <span class="n">base</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># noinspection PyPep8Naming</span>
<span class="kn">import</span> <span class="nn">psycopg</span> <span class="k">as</span> <span class="nn">Database</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="c1"># noinspection PyPep8Naming</span>
<span class="kn">import</span> <span class="nn">psycopg2</span> <span class="k">as</span> <span class="nn">Database</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ImproperlyConfigured</span><span class="p">(</span><span class="s2">"Error loading psycopg2 or psycopg module"</span><span class="p">)</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">DatabaseCredentials</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">settings_dict</span><span class="p">:</span> <span class="nb">dict</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">creds</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"init vault credentials"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">credential_name</span> <span class="o">=</span> <span class="n">settings_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"VAULT_ROLE"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">static_credential_name</span> <span class="o">=</span> <span class="n">settings_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"VAULT_STATIC_ROLE"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mount_point</span> <span class="o">=</span> <span class="n">settings_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"VAULT_MOUNT_POINT"</span><span class="p">,</span> <span class="s2">"database"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">vault_url</span> <span class="o">=</span> <span class="n">settings_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"VAULT_ADDR"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">vault_token</span> <span class="o">=</span> <span class="n">settings_dict</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"VAULT_TOKEN"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">client</span> <span class="o">=</span> <span class="n">hvac</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">vault_url</span><span class="p">,</span> <span class="n">token</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">vault_token</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">refresh_now</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">get_conn_params_from_vault</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_params</span><span class="p">):</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">"user"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">creds</span><span class="p">[</span><span class="s2">"username"</span><span class="p">]</span>
<span class="n">conn_params</span><span class="p">[</span><span class="s2">"password"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">creds</span><span class="p">[</span><span class="s2">"password"</span><span class="p">]</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">"Getting db creds: user=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">creds</span><span class="p">[</span><span class="s1">'username'</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">refresh_now</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">"refreshing credentials for </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">credential_name</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">static_credential_name</span><span class="p">:</span>
<span class="n">our_creds</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">secrets</span><span class="o">.</span><span class="n">database</span><span class="o">.</span><span class="n">get_static_credentials</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">credential_name</span><span class="p">,</span> <span class="n">mount_point</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mount_point</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">our_creds</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">secrets</span><span class="o">.</span><span class="n">database</span><span class="o">.</span><span class="n">generate_credentials</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">credential_name</span><span class="p">,</span> <span class="n">mount_point</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mount_point</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">creds</span> <span class="o">=</span> <span class="n">our_creds</span><span class="p">[</span><span class="s2">"data"</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">DatabaseWrapper</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">DatabaseWrapper</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">settings_dict</span><span class="p">,</span> <span class="n">alias</span><span class="o">=</span><span class="n">DEFAULT_DB_ALIAS</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">database_credentials</span> <span class="o">=</span> <span class="n">DatabaseCredentials</span><span class="p">(</span><span class="n">settings_dict</span><span class="p">)</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">settings_dict</span><span class="p">,</span> <span class="n">alias</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_new_connection</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_params</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"get connection"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">database_credentials</span><span class="o">.</span><span class="n">get_conn_params_from_vault</span><span class="p">(</span><span class="n">conn_params</span><span class="p">)</span>
<span class="n">conn</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">DatabaseWrapper</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_new_connection</span><span class="p">(</span><span class="n">conn_params</span><span class="p">)</span>
<span class="k">return</span> <span class="n">conn</span>
<span class="k">except</span> <span class="n">Database</span><span class="o">.</span><span class="n">OperationalError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="c1"># there doesn't appear to be a good way to check for a specific error</span>
<span class="c1"># other than to read the string and look for "authentication failed"</span>
<span class="k">if</span> <span class="s2">"authentication failed"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="k">raise</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Authentication error. Going to refresh secret and try again."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">database_credentials</span><span class="o">.</span><span class="n">refresh_now</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">database_credentials</span><span class="o">.</span><span class="n">get_conn_params_from_vault</span><span class="p">(</span><span class="n">conn_params</span><span class="p">)</span>
<span class="n">conn</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">DatabaseWrapper</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_new_connection</span><span class="p">(</span><span class="n">conn_params</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s2">"Successfully refreshed secret and established new database connection."</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">conn</span>
</code></pre></div>
Kubernetes Load Balancer Reset2023-10-07T10:50:00-04:002023-10-07T10:50:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-07:/kubernetes-load-balancer-reset.html<p>This morning I had the need to change the IP address configuration for
the load balancer in our k8s cluster. The basics of changing the main
pool in <a href="https://metallb.universe.tf/">metallb</a> were straightforward
enough, but when I applied my changes, I didn't get what I needed.</p>
<p>So, what happened? Originally, I wasn't …</p><p>This morning I had the need to change the IP address configuration for
the load balancer in our k8s cluster. The basics of changing the main
pool in <a href="https://metallb.universe.tf/">metallb</a> were straightforward
enough, but when I applied my changes, I didn't get what I needed.</p>
<p>So, what happened? Originally, I wasn't thinking like a kubernetes cluster,
so I'd not realized that the load balancer itself drives
addressing. As such, I was focussed on things like restarting the pods that
had been assigned addresses. This did no good, although I did learn
how to force <a href="https://fluxcd.io">fluxcd</a> to
<a href="https://fluxcd.io/flux/installation/configuration/helm-drift-detection/">detect and correct helm chart drift</a>
which was super useful when I accidentally deleted a deployment that
didn't come back.</p>
<p>Once I realized that <code>metallb</code> is a kubernetes <em>operator</em> that looks at
the <code>Service</code> CRD for <em>type</em> <code>LoadBalancer</code> and plumbs the address and
network for those services, I needed to focus on the <code>metallb</code> controller
pod, as that was what wasn't effecting the changes.</p>
<p>I looked at the logs and realized it would not pick up the new
configuration because there were IP addresses in the old range in use.
It turns out that there's no way to automatically update these, but
if you restart your controller deployment:</p>
<div class="codehilite"><pre><span></span><code>kubectl<span class="w"> </span>rollout<span class="w"> </span>restart<span class="w"> </span>deployment<span class="w"> </span>metallb-controller<span class="w"> </span>-n<span class="w"> </span>metallb-system
</code></pre></div>
<p>Then it will pick up the new configuration and assign new addresses to
any service already in use.</p>
Recovering longhorn backups2023-10-07T10:50:00-04:002023-10-07T10:50:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-10-07:/recovering-longhorn-backups.html<p>Another chapter in my learning kubernetes the hard way, this time Longhorn.</p>
<p>Probably ill-advisedly, I'm using ephemeral volumes for my storage
volumes in Longhorn <em>and</em> have a habit of leaving the nodes in the
cluster as they're being rebuilt. Generally, this isn't a problem. This weekend,
I was a bit …</p><p>Another chapter in my learning kubernetes the hard way, this time Longhorn.</p>
<p>Probably ill-advisedly, I'm using ephemeral volumes for my storage
volumes in Longhorn <em>and</em> have a habit of leaving the nodes in the
cluster as they're being rebuilt. Generally, this isn't a problem. This weekend,
I was a bit too cavalier about handling the rebuild process and
didn't prune the temporary volumes each time I replaced a storage node,
resulting in all of my storage getting nuked.</p>
<p>In my case, even the "persistent" volumes are all just cache, so it didn't
really matter. However, since I'm also backing this up to an S3-compatible
storage system, it gave me an opportunity to try retrieval.</p>
<h2>Ephemeral storage in persistent nodes</h2>
<p>If I were to remove nodes completely and bring them up afresh, I wouldn't have
these problems. However, I've had the practice of draining/cordoning nodes and
then rebuilding them and re-establishing them in the cluster without removing them
completely from the cluster.</p>
<p>This is marginally faster, but results in the system thinking it was just
temporarily disconnected instead of completely gone. Because of this, the
longhorn storage expects the volumes to be there and present. The current failure
mode is for them to indicate an error, but not allocate new space. This makes
sense in terms of restoring from backups; but it's not helpful in my case, since it
takes up the volume slot and prevents the other nodes from rebuilding.</p>
<p>This weekend, I rebuilt all 4 of my storage nodes, resulting in a complete loss
of data. Configuration was, of course, fine, since that's in the etcd (which I
didn't screw up this time).</p>
<h2>Restoring the pvc</h2>
<p>As an experiment, I wanted to try restoring the backups from my S3-like backup
storage to see if it would work. This is the process that worked for me:</p>
<ol>
<li>
<p>Quiesce the dependent pod by scaling down to zero:</p>
<div class="codehilite"><pre><span></span><code>kubectl scale deploy --replicas=0 renovate-whitesource-renovate
</code></pre></div>
</li>
<li>
<p>Use the GUI to restore the backup to a new volume (named appropriately). In
my case, I named it <code>mend-restored</code></p>
</li>
<li>
<p>Wait for the restore to finish</p>
</li>
<li>
<p>Delete the old Volume (GUI)</p>
</li>
<li>
<p>Create PV/PVC on the backup (GUI). Use the existing name for the PVC.</p>
</li>
<li>
<p>Once the PV and PVC are available, scale the dependent pod back up:</p>
<div class="codehilite"><pre><span></span><code>kubectl scale deploy --replicas=1 renovate-whitesource-renovate
</code></pre></div>
</li>
</ol>
Vaulting Database Credentials2023-09-25T07:06:00-04:002023-09-25T07:06:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-09-25:/vaulting-database-credentials.html<p>Over the past year, I've been experimenting with
<a href="https://www.hashicorp.com">Hashicorp</a>
<a href="https://www.hashicorp.com/products/vault">Vault</a>, using the
open-source/community version for some internal experiments, including
some with high availability.</p>
<p>In a separate article, I'll go over a
<a href="https://www.gaige.net/vault-local-testing-setup.html">test configuration of Vault</a>,
but all of the notes here are agnostic to the use of HCP
(Hashicorp's …</p><p>Over the past year, I've been experimenting with
<a href="https://www.hashicorp.com">Hashicorp</a>
<a href="https://www.hashicorp.com/products/vault">Vault</a>, using the
open-source/community version for some internal experiments, including
some with high availability.</p>
<p>In a separate article, I'll go over a
<a href="https://www.gaige.net/vault-local-testing-setup.html">test configuration of Vault</a>,
but all of the notes here are agnostic to the use of HCP
(Hashicorp's cloud services) or a private instance.</p>
<h2>Setting up database connections</h2>
<p>Contrary to what I originally thought, you only need to set up a single database
configuration for each database server/cluster. (You can create more if you need
to silo your controls further, but doing so can make the role map unnecessarily
confusing). Specifically, the <code>connection_url</code> from the database configuration
does not limit database access to the roles created through or administered by
the vault database connection.</p>
<p>In most cases, set up a single database config to your <code>postgres</code> database and
use it as the connection for all of your static and dynamic roles.</p>
<p>I'll walk through setting up a local vault and database connection
for a test scenario in
<a href="https://www.gaige.net/vault-local-testing-setup.html">Local Testing</a>.</p>
<h3>Vault connection user</h3>
<p>You'll definitely want to create a separate admin user for use by
vault in managing credentials. Once you configure your connection with this
user and rotate the password, you'll not have access to that password and
as such it won't be available for break-glass or other administrative
duties.</p>
<p>Recommendations from the internet are that the vault admin user just be
able to provision users (<code>CREATEROLE</code> privilege can grant membership
in other roles), and then create a postgresql role which can be
used (assumed) by the vault role in the event that it needs to create
database objects that have ownership (tables, sequences, etc.).
Your specific use case may vary, but least privilege would be
the goal with multiple roles.</p>
<h3>Static vs dynamic roles</h3>
<p>Vault has two types of roles for accessing databases, static and dynamic. The
static roles have user-specified usernames, whereas the dynamic roles create
a new (usually short-lived) user while being used. The biggest difference is
that the dynamic roles are deleted once they're no longer in use (see caveat
below on ownership).</p>
<p>A static role can adopt a role already in a database and is automatically
rotated after a specified period of time.</p>
<p>A dynamic role is a unique user id that lasts for a shorter period
of time, with extensions possible through extending the lease, and are
deleted after their expiration time. Generally speaking, this is the target
state, as they provide isolationa dn limit usefulness of leaked credentials.</p>
<h3><a id="ownership-and-roles"></a>Ownership and roles</h3>
<p>One problem with dynamic roles comes into play when creating objects. Quoting from
the postgresql
<a href="https://www.postgresql.org/docs/current/role-removal.html">documentation on DROP ROLE</a>:</p>
<blockquote>
<p>Because roles can own database objects and can hold privileges to
access other objects, dropping a role is often not just a matter
of a quick DROP ROLE. Any objects owned by the role must first be
dropped or reassigned to other owners; and any permissions granted
to the role must be revoked.</p>
</blockquote>
<p>So, although the dynamic roles give us the safety we desire, they do create some
complications when objects may be owned by those roles. To solve for this, we add
some complexity, by creating a new role to hold the priviliges, then give the
dynamic roles
<a href="https://www.postgresql.org/docs/current/role-membership.html#ROLE-MEMBERSHIP">role membership</a>
in that role, and finally we assume the role for each database session, ensuring
that we are acting on behalf of the role we assume, including its priviliges and
ownership.</p>
<p>Creating the roles may take a little time, but it has a couple of other nice side
effects, including placing the determination of specific access to the database
instead of hard-coding that into your vault configuration.</p>
<p>Further reading:</p>
<ul>
<li><a href="https://www.gaige.net/postgres-roles-and-privileges.html">Postres roles and priviliges</a>
for details on how I designed my roles.</li>
<li><a href="https://www.gaige.net/django-and-vault.html">Django and vault</a>
describes how I integrated vault into my django code.</li>
<li><a href="https://www.gaige.net/vault-local-testing-setup.html">Vault local testing setup</a>
demonstrates creating a local testing setup for vault and postgres</li>
</ul>
Kubernetes etcd near disaster2023-07-30T07:06:00-04:002023-07-30T07:06:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-07-30:/kubernetes-etcd-near-disaster.html<p>This post is mostly a warning to me for the future, but hopefully it'll prevent
somebody else from going through the same problem. I've been running a small
Kubernetes cluster for a couple of years now, mostly as an experiment and to
keep my skills tuned for new tooling. Part …</p><p>This post is mostly a warning to me for the future, but hopefully it'll prevent
somebody else from going through the same problem. I've been running a small
Kubernetes cluster for a couple of years now, mostly as an experiment and to
keep my skills tuned for new tooling. Part of that has been making sure I use
reasonable tooling and automate as much as I can.</p>
<p>So far, I've been pretty happy using:</p>
<ul>
<li><a href="https://rke.docs.rancher.com">rke</a></li>
<li><a href="https://docs.gitlab.com/ee/user/clusters/agent/install/">GitLab CI/CD Workflow</a></li>
<li><a href="https://metallb.universe.tf">metallb</a></li>
</ul>
<h2>Keeping kubernetes up to date</h2>
<p>I've been in the habit of keeping my k8s cluster up-to-date for some time.
Usually just redeploying the RKE cluster whenever there's a notable upgrade
and keeping major dependencies up to date using automation based on
<a href="https://www.mend.io">mend</a>.</p>
<p>This has worked well for the kubernetes infrastructure, but since the "machines"
that run my cluster are all VMs, I also want to update them occasionally, updating
the underlying OS and proving that I can rebuild the environment.</p>
<p>For worker nodes in the cluster, this has generally worked well. I have a basic,
automated by Ansible process of:</p>
<ol>
<li>Set the node to unschedulable</li>
<li>replace the node</li>
<li>run <code>rke</code> command in order to bring the node back into the cluster</li>
</ol>
<p>This has worked fine, probably because the worker nodes, once quiesced, are
not unique in any way.</p>
<h1>Updating my control plane nodes</h1>
<p>The problem came when I went to follow the same procedure for my control
plane nodes. I took the first node offline, built a new node, and ran
<code>rke</code> to bring the node back online. Everything seemed to be functioning
well, so I went on to the next node and everything came to a halt.</p>
<p>It took me a few minutes to figure out what I'd done, but the key is that,
unlike the worker nodes, the nodes running etcd are a bit special. In
particular, they carry a unique ID that is embedded in their local database.</p>
<p>Running <code>docker exec etcd etcdctl --write-out=table member list</code>, you can see
the ID on the left:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">+------------------+---------+-------------+--------------------------+--------------------------+------------+</span>
<span class="c">| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |</span>
<span class="nb">+------------------+---------+-------------+--------------------------+--------------------------+------------+</span>
<span class="c">| 51e442e065ed8da9 | started | etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">1 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">1:2380 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">1:2379 | false |</span>
<span class="c">| 7c17ab818595f4fe | started | etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">0 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">0:2380 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">0:2379 | false |</span>
<span class="c">| d085086f6d909371 | started | etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">2 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">2:2380 | https://etcd</span><span class="nb">-</span><span class="c">node</span><span class="nb">-</span><span class="c">2:2379 | false |</span>
<span class="nb">+------------------+---------+-------------+--------------------------+--------------------------+------------+</span>
</code></pre></div>
<p>Everything was fine when I deleted the first node, since the environment was
configured for HA, it had 2/3 of its nodes available, which meant it still
had a good voting majority left. And, when the node was re-provisioned, it
was still fine, because it could contact 3/4 of the etcd nodes. However, when
I went to replace the second node, the entire cluster failed, because etcd
reached a state where only 2/4 of the etcd nodes were available and it couldn't reach quorum.</p>
<p>Once I was able to ascertain that the original node hadn't been replaced in
the node list, the solution was relatively simple, and I deleted the errant
node using:</p>
<div class="codehilite"><pre><span></span><code>docker<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>etcd<span class="w"> </span>etcdctl<span class="w"> </span>member<span class="w"> </span>remove<span class="w"> </span><span class="o">[</span>id<span class="o">]</span>
</code></pre></div>
<p>In the future, when working with the etcd nodes, I've made modifications to
my MOP and the my ansible scripting:</p>
<ol>
<li>
<p>Make sure to check the status of the etcd membership before replacing
any node</p>
<p><code>docker exec etcd etcdctl --write-out=table member list</code></p>
</li>
<li>
<p>Make an explicit snapshot of the cluster before replacing the nodes:</p>
<p><code>rke etcd snapshot-save --name extra_snapshot.db --config cluster.yml</code></p>
</li>
<li>
<p>Remove the old etcd node as soon as possible to prevent negative effect on the quorum:</p>
<p><code>docker exec etcd etcdctl member remove [id]</code></p>
</li>
</ol>
<h2>Recovery from failure</h2>
<p>In order to get the cluster back into working shape, I did do a restore and
rebuild once I figured out what had gone on. This also involved using the
most recent backup from etcd. (I also took a backup of the botched etcd
situation before restoring).</p>
<div class="codehilite"><pre><span></span><code>rke<span class="w"> </span>etcd<span class="w"> </span>snapshot-save<span class="w"> </span>--name<span class="w"> </span>disaster.db<span class="w"> </span>--config<span class="w"> </span>cluster.yml
rke<span class="w"> </span>etcd<span class="w"> </span>snapshot-restore<span class="w"> </span>--name<span class="w"> </span>extra_snapshot.db<span class="w"> </span>--config<span class="w"> </span>cluster.yml
rke<span class="w"> </span>up
</code></pre></div>
<p>Note that there is reasonable
<a href="https://etcd.io/docs/v2.3/admin_guide/#disaster-recovery">disaster recovery</a>
documentation in etcd's documentation.</p>
Elastic index correction2023-07-03T07:04:00-04:002023-07-03T07:04:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-07-03:/elastic-index-correction.html<p>Recently, I noticed a problem with my Index Lifecycle Management (ILM) not appropriately
rotating indexes. The error was not super clear, but I did notice that the existing
index had just reached 90 days without closing and that was the first move in the
ILM. It was clear that the …</p><p>Recently, I noticed a problem with my Index Lifecycle Management (ILM) not appropriately
rotating indexes. The error was not super clear, but I did notice that the existing
index had just reached 90 days without closing and that was the first move in the
ILM. It was clear that the 30-day rollover wasn't happening.</p>
<p>The primary problem was easy to solve, which was to make sure that the write index
was set correctly and the index was attached to the template with the alias:</p>
<div class="codehilite"><pre><span></span><code><span class="err">PUT</span><span class="w"> </span><span class="kc">f</span><span class="err">ilebea</span><span class="kc">t</span><span class="mf">-8.0.0-2023-04-01-000001</span>
<span class="p">{</span>
<span class="w"> </span><span class="nt">"aliases"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"filebeat-8.0.0"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"is_write_index"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>That resolved part of the problem, but the roll-over then occurred and it created
<code>filebeat-8.0.0-2023-04-01-000002</code>, which definitely wasn't what I wanted (although
in truth that date is just for the humans, the ILM uses the write dates).</p>
<p>To fix this, I needed to:</p>
<ol>
<li>
<p>Stop the ILM</p>
<p><code>POST _ilm/stop</code></p>
</li>
<li>
<p>Create a new index using the date fields</p>
<p><code>PUT %3Cfilebeat-8.0.0-%7Bnow%2Fd%7D-000001%3E</code></p>
</li>
<li>
<p>Set the write index correctly for both indexes:</p>
<p><code>POST /_aliases</code></p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="nt">"actions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"add"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"filebeat-8.0.0-2023-07-03-000001"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"alias"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"filebeat-8.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"is_write_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"add"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"filebeat-8.0.0-2023-04-01-000002"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"alias"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"filebeat-8.0.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"is_write_index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
</li>
<li>
<p>Turn ILM back on</p>
<p><code>POST _ilm/start</code></p>
</li>
</ol>
<p>Relatively straightforward. The only hiccup was that the interim index was now out of
sync with the ILM program, showing:</p>
<div class="codehilite"><pre><span></span><code><span class="n">illegal_argument_exception</span><span class="o">:</span><span class="w"> </span><span class="n">index</span><span class="w"> </span><span class="o">[</span><span class="n">filebeat</span><span class="o">-</span><span class="mf">8.0</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="mi">2023</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">000002</span><span class="o">]</span><span class="w"> </span><span class="k">is</span><span class="w"> </span><span class="n">not</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">write</span><span class="w"> </span><span class="n">index</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">alias</span><span class="w"> </span><span class="o">[</span><span class="n">filebeat</span><span class="o">-</span><span class="mf">8.0</span><span class="o">.</span><span class="mi">0</span><span class="o">]</span>
</code></pre></div>
<p>Since the temporary rollover index was small and didn't contain anything essential,
I decided to delete it. There were some postings for older versions of ES that suggested
ways of fixing this, but with 8+ they didn't seem to work.</p>
<p>Also of note: last year, I described
<a href="https://www.gaige.net/renaming-elasticsearch-indexes.html">Renaming Elasticsearch indexes</a>
when the situation arose to change the name of an index template.</p>
Poetry in GitLab2023-05-14T12:31:00-04:002023-05-14T12:31:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-05-14:/poetry-in-gitlab.html<p>This weekend, I had occasion to build a new python-based utility and
leaned in to my existing poetry tooling in order to do so. While starting
the new project, I wanted to take advantage of some gitlab automation I'd
previously used on other projects, so I figured I'd document it …</p><p>This weekend, I had occasion to build a new python-based utility and
leaned in to my existing poetry tooling in order to do so. While starting
the new project, I wanted to take advantage of some gitlab automation I'd
previously used on other projects, so I figured I'd document it here.</p>
<h2>Tooling overview for automation</h2>
<p>The purpose of the gitlab automation here is to go from a feature
branch to a new release without having to do any of the work myself.</p>
<p>I'm using a bunch of tools to achieve this:</p>
<ul>
<li><a href="https://python-poetry.org/docs/"><code>poetry</code></a> for dependency management and packaging</li>
<li><a href="https://commitizen-tools.github.io/commitizen/"><code>commitizen</code></a> for enforcing
<a href="https://www.conventionalcommits.org/en/v1.0.0/">conventional commits</a> and
managing release notes</li>
<li><a href="https://docs.pytest.org/en/7.3.x/"><code>pytest</code></a> for test running and reporting</li>
<li><a href="https://tox.wiki/en/latest/"><code>tox</code></a> for test automation in multiple language
versions (currently 3.10 and 3.11)</li>
</ul>
<p>And, for good measure, I'll mention <a href="https://gitlab.com">GitLab</a>
(the Pro version) for source repository and CI/CD,
and <a href="https://www.jetbrains.com/pycharm/features/">JetBrains PyCharm</a>, which I use
as my IDE most of the time.</p>
<h2>Automating the poetry delivery pipeline</h2>
<p>Once I've got the project building and tests running, then I want to start
rolling it out in versions. I first established this pipeline for another
command-line tool (<code>certalerter</code>, my alerting tool for <code>certlogger</code>), so
adapting for a new project should be straightforward.</p>
<p>I'm going to elide the coding and testing and stick to the automation for this post,
and mostly do it by going through my <code>.gitlab-ci.yml</code> file a bit at a time.</p>
<h3>Overall workflow</h3>
<p>I've broken the workflow into 5 stages and limited the running of the
workflow to: merge requests, commits to main branch when not in a merge
request, adding a tag (mostly to handle releases).</p>
<p>I'm running all of this in docker (possibly k8s, but I haven't specifically
enabled that yet). My python is pretty clean and I havne't had any problems
with portability.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">workflow</span><span class="p">:</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="s">'$CI_PIPELINE_SOURCE</span><span class="nv"> </span><span class="s">==</span><span class="nv"> </span><span class="s">"merge_request_event"'</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="s">'$CI_COMMIT_BRANCH</span><span class="nv"> </span><span class="s">&&</span><span class="nv"> </span><span class="s">$CI_OPEN_MERGE_REQUESTS'</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">never</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="s">'$CI_COMMIT_BRANCH'</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_TAG</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span>
<span class="nt">default</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python:3.11</span>
</code></pre></div>
<p>There are 5 stages, of which most of them are pretty straightforward</p>
<div class="codehilite"><pre><span></span><code><span class="nt">stages</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bump</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">package</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">release</span>
</code></pre></div>
<h3>Building the project</h3>
<p>Building the project is pretty straightforward, load in <code>poetry</code> to
get our environment and the let it <code>build</code>. I've chosen to capture
the distribution binaries (<code>whl</code> and <code>tar.gz</code> files) in the artifacts
paths so that they don't need to be rebuilt for the testing phase. I'm
not using the <code>PyPi</code> repository from gitlab yet, because I don't want
every build to be uniquely kept there, but that's addressed in the
<code>package</code> phase later.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">build-job</span><span class="p">:</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip install poetry</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry build</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dist/ct_nagios_plugins*.whl</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dist/ct_nagios_plugins*.tar.gz</span>
<span class="w"> </span><span class="nt">expire_in</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1 week</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</code></pre></div>
<h3>Testing and coverage</h3>
<p>In order to enable pushing automatically to production, I feel it's necessary
to have well maintained test suites. As such, code is tested on every commit
(and in all major environments) and coverage is maintained to see when the
tests are back-sliding.</p>
<p>Each of the test environments is started in the appropriate python image
and then <code>tox</code> and the <code>coverage</code> tools are installed so that we don't
need to fully install python. Since the goal here is to create a stand-alone
package, I want to take care not to introduce any unintended <code>poetry</code>
dependencies.</p>
<p>The funky <code>grep</code>/<code>sed</code>/<code>awk</code> bit is to tease the coverage out of the
coverage file for use by gitlab. The <code>|| true</code> at the end of it ensures
that being unable to get coverage through this method doesn't spoil the
stage.</p>
<p>Finally, the test logs (<code>junit-*.xml</code>) and coverage reports (<code>coverage-*.xml</code>)
are stored as artifacts.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">test</span><span class="p">:</span>
<span class="w"> </span><span class="nt">needs</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build-job</span>
<span class="w"> </span><span class="nt">parallel</span><span class="p">:</span>
<span class="w"> </span><span class="nt">matrix</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">PYTHON_VERSION</span><span class="p">:</span><span class="w"> </span><span class="s">"3.11"</span>
<span class="w"> </span><span class="nt">TOXENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">py311</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">PYTHON_VERSION</span><span class="p">:</span><span class="w"> </span><span class="s">"3.10"</span>
<span class="w"> </span><span class="nt">TOXENV</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">py310</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python:${PYTHON_VERSION}</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip install tox coverage</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">tox --installpkg dist/*.whl</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage xml -o coverage-${PYTHON_VERSION}.xml</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">></span>
<span class="w"> </span><span class="no">grep ^\<coverage coverage-${PYTHON_VERSION}.xml</span>
<span class="w"> </span><span class="no">| sed -n -e 's/.*line-rate=\"\([0-9.]*\)\".*/\1/p'</span>
<span class="w"> </span><span class="no">| awk '{print "CodeCoverageOverall =" $1*100}'</span>
<span class="w"> </span><span class="no">|| true</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">reports</span><span class="p">:</span>
<span class="w"> </span><span class="nt">junit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">junit-*.xml</span>
<span class="w"> </span><span class="nt">coverage_report</span><span class="p">:</span>
<span class="w"> </span><span class="nt">coverage_format</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cobertura</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage-*.xml</span>
<span class="w"> </span><span class="nt">coverage</span><span class="p">:</span><span class="w"> </span><span class="s">'/^CodeCoverageOverall</span><span class="nv"> </span><span class="s">=(\d+\.\d+)$/'</span>
</code></pre></div>
<h3>Bumping versions</h3>
<p>In addition to the previously-mentioned rules for running, the version bump
is very selective. It will only run on commits to <code>main</code> where <code>bump</code> is not
part of the commit message. This (hopefully) prevents it from running twice
without need. It also should stop loops.</p>
<p>Note the use of <code>CI_BUMP_TOKEN</code> here, which is a Personal Access Token (PAT)
for GitLab that has permissions to <code>read_repository</code> and <code>write_repository</code>
so that it can be used to write back to the repo. When I tried this originally,
I expected to be able to commit back to my own repo, but ran into trouble, so
using the PAT here makes that straightforward. The <code>CI_BUMP_GITLAB_ID</code> is
probably not necessary, as <code>__token__</code> should suffice.</p>
<p>Using <code>poetry</code> and <code>cz</code> here guarantees that the steps that are expected all
run, but it also results in the above requirements. If I weren't committing
back, but just setting a tag or release, I could easily do that with the API.
Specifically, the <code>CI_JOB_TOKEN</code> doesn't have <code>write_repository</code> permission.</p>
<p>In my case, I use a specific <em>PAT</em> to this repository, so that I can limit
the blast radius. I'd be happier if there were a way to request a read/write
<code>CI_JOB_TOKEN</code> for certain stages, but even if that were available, it's
not clear how that would be governed effectively without giving all
stages in the pipeline access.</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># need to clean in case tagging is screwy, since `git clean` doesn't know to remove tags</span>
<span class="nt">bump</span><span class="p">:</span>
<span class="w"> </span><span class="nt">needs</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bump</span>
<span class="w"> </span><span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">GIT_STRATEGY</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">clone</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip install poetry</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry install</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">git config --global user.email "${GITLAB_USER_EMAIL}"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">git config --global user.name "${GITLAB_USER_NAME}"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">exit_code=0</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry run cz bump --annotated-tag --changelog || exit_code=$?</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo "$exit_code is exit code ; $? was result"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">if [ $exit_code -eq 0 ]</span>
<span class="w"> </span><span class="no">then</span>
<span class="w"> </span><span class="no">git remote set-url origin ${CI_SERVER_PROTOCOL}://${CI_BUMP_GITLAB_ID}:${CI_BUMP_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}</span>
<span class="w"> </span><span class="no">git push origin --follow-tags HEAD:${CI_COMMIT_BRANCH}</span>
<span class="w"> </span><span class="no">elif [ $exit_code -eq 21 ]</span>
<span class="w"> </span><span class="no">then</span>
<span class="w"> </span><span class="no">echo "Skipping push with no version change"</span>
<span class="w"> </span><span class="no">elif [ $exit_code -eq 3 ]</span>
<span class="w"> </span><span class="no">then</span>
<span class="w"> </span><span class="no">echo "Skipping push with no commits"</span>
<span class="w"> </span><span class="no">else</span>
<span class="w"> </span><span class="no">echo "cz error code $exit_code"</span>
<span class="w"> </span><span class="no">exit $exit_code</span>
<span class="w"> </span><span class="no">fi</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_BRANCH==$CI_DEFAULT_BRANCH && $CI_COMMIT_TITLE =~ /^bump/</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">never</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_BRANCH==$CI_DEFAULT_BRANCH</span>
<span class="c1"># skip on bump, because you'll never bump after bump</span>
</code></pre></div>
<h3>Packaging the job</h3>
<p>As with the <code>bump</code> stage, the <code>package</code> stage runs only at specific
times. In particular, it will only run directly following a <code>bump</code> commit on
the <code>main</code> branch.</p>
<p>Theoretically, I could use <code>poetry</code> and then make use of the <code>publish</code>
command, but in this case, <code>twine</code> is fine (and dedicated).</p>
<div class="codehilite"><pre><span></span><code><span class="nt">package-job</span><span class="p">:</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">package</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip install twine</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --verbose --disable-progress-bar --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_BRANCH==$CI_DEFAULT_BRANCH && $CI_COMMIT_TITLE =~ /^bump/</span>
</code></pre></div>
<h3>Finishing off the release</h3>
<p>The final phase, which only happens on tagged commits, is to tag the release. GitLab
makes this <a href="https://docs.gitlab.com/ee/user/project/releases/release_cicd_examples.html#create-a-release-when-a-git-tag-is-created">easy</a>
by directly supporting the release process in the CI file.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">release_job</span><span class="p">:</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">release</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">registry.gitlab.com/gitlab-org/release-cli:latest</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_TAG</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo "Running the release job for $CI_COMMIT_TAG."</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"awk</span><span class="nv"> </span><span class="s">'/^##</span><span class="nv"> </span><span class="s">Unreleased/</span><span class="nv"> </span><span class="s">{</span><span class="nv"> </span><span class="s">next</span><span class="nv"> </span><span class="s">}</span><span class="nv"> </span><span class="s">;</span><span class="nv"> </span><span class="s">/^##</span><span class="nv"> </span><span class="s">/</span><span class="nv"> </span><span class="s">{</span><span class="nv"> </span><span class="s">r++</span><span class="nv"> </span><span class="s">;</span><span class="nv"> </span><span class="s">if</span><span class="nv"> </span><span class="s">(</span><span class="nv"> </span><span class="s">r</span><span class="nv"> </span><span class="s"><2)</span><span class="nv"> </span><span class="s">{</span><span class="nv"> </span><span class="s">print</span><span class="nv"> </span><span class="s">;</span><span class="nv"> </span><span class="s">next</span><span class="nv"> </span><span class="s">}</span><span class="nv"> </span><span class="s">else</span><span class="nv"> </span><span class="s">{</span><span class="nv"> </span><span class="s">exit</span><span class="nv"> </span><span class="s">}</span><span class="nv"> </span><span class="s">};</span><span class="nv"> </span><span class="s">/^/</span><span class="nv"> </span><span class="s">{</span><span class="nv"> </span><span class="s">print</span><span class="nv"> </span><span class="s">}</span><span class="nv"> </span><span class="s">;'</span><span class="nv"> </span><span class="s"><</span><span class="nv"> </span><span class="s">CHANGELOG.md</span><span class="nv"> </span><span class="s">>INCREMENTAL_CHANGELOG.md"</span>
<span class="w"> </span><span class="nt">release</span><span class="p">:</span>
<span class="w"> </span><span class="nt">tag_name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_TAG</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s">'v$CI_COMMIT_TAG'</span>
<span class="w"> </span><span class="nt">description</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">INCREMENTAL_CHANGELOG.md</span>
</code></pre></div>
Monitor fleet aging2023-05-14T10:40:00-04:002023-05-14T10:40:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-05-14:/monitor-fleet-aging.html<h2>Background</h2>
<p>Generally speaking, I refresh most of my systems pretty regularly, spurred on
by security concerns, general hygeine, a desire to make sure the automation
doesn't age out, and certificate expiration.</p>
<p>Although I don't need to refersh systems due to certificate expiration, it
has historically been the easiest indicator of …</p><h2>Background</h2>
<p>Generally speaking, I refresh most of my systems pretty regularly, spurred on
by security concerns, general hygeine, a desire to make sure the automation
doesn't age out, and certificate expiration.</p>
<p>Although I don't need to refersh systems due to certificate expiration, it
has historically been the easiest indicator of systems that are getting a
little long in the tooth.</p>
<p>Working on some systems this weekend, I noticed some out-of-date copies of
postgresql...really out of date..like close to a year old. This is what
sent me off on this weekend's adventure.</p>
<h2>What do you mean by refresh and why?</h2>
<p>Given our penchant or building everything using Ansible, when I indicate
I'm refreshing a system, that means the old VM gets taken down and a new one
is built to then-current specifications as a replacement.</p>
<p>Rob and I have nurtured this workflow for years (ever since moving to using
<a href="https://docs.ansible.com">ansible</a> for automation). In all cases, I build
staging environments before production and in most cases there are some
reasonable automated tests for that process.</p>
<p>As to why? The answer is mostly one of convenience, although there are
security arguments as well, both getting the latest versions of libraries
that may contain vulnerabilities and dislodging anything bad that may be
sitting on the virtual machines.</p>
<h2>Monitoring the fleet age</h2>
<p>Based on the recent discovery of some aging systems, I figured that I
should find a way to add this process to our monitoring system, the
venerable <a href="https://www.nagios.org">Nagios</a>.</p>
<p>This didn't need to be particularly complex, but I needed the nagios
server to reach out to the SmartOS Global Zones in order to get information
about the running VMs. Historically, we've done with with captive SSH, using
dedicated keys and lines in <code>~/.ssh/authorized_keys</code> which take advantage
of the <code>command=</code> command in order to run a program, potentially with
information from the incoming SSH connection. Results are sent in text,
but preferably encoded in JSON or similar.</p>
<h3>a new python framework for ssh requests</h3>
<p>Most of our previous commands piggy-backed on the <code>check_by_ssh</code> checker,
which is a standard nagios plugin. However, that command assumes that we
put all of the intelligence at the other end of the line (on the recipient)
and basically run the checks there. That could be done, but the need to do
date math made coming up with an appropirate one-liner a bit ridiculous,
so I decided to go with python.</p>
<p>The python code was strightforward, and I used my existing <code>poetry</code>-based
environment as a starting point, creating a couple of new commands which
I'd install on the nagios servers: one for SmartOS and another for AWS.</p>
<p>By making use of my existing <code>poetry</code> workflows, I got a number of things for
free, including updating release notes, packaging releases in gitlab, etc.</p>
<h3>Integrating with nagios</h3>
<p>The <code>nagios</code> integration should have been simple, but for one small issue:
I needed to parameterize the global zone system so that the command could
take place there.</p>
<p>After some digging through the
<a href="https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/objectdefinitions.html#service">documentation for nagios</a>,
I found the section on <a href="https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/macros.html">custom macro variables</a>,
which is exactly what I needed in this case. I wanted to add a new variable
<code>_GZHOST</code> to my existing host definitions which would indicate which host to
query about the underlying VM. I already had this infromation in the <code>PARENTS</code>
field, which I thought I could use as <code>$HOSTPARENTS$</code>, but it turns out that
for some reason that's not exposed.</p>
<p>In this case, I was able to use <code>$_HOSTGZHOST</code> in my <code>command</code> definition in
<code>commands.cfg</code>, resulting in:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">define</span><span class="w"> </span><span class="nv">command</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nv">command_name</span><span class="w"> </span><span class="nv">check_smartos_vm_age</span>
<span class="w"> </span><span class="nv">command_line</span><span class="w"> </span><span class="o">/</span><span class="nv">opt</span><span class="o">/</span><span class="nv">local</span><span class="o">/</span><span class="nv">bin</span><span class="o">/</span><span class="nv">ct</span><span class="o">-</span><span class="nv">smartos</span><span class="o">-</span><span class="nv">vm</span><span class="w"> </span><span class="o">-</span><span class="nv">H</span><span class="w"> </span><span class="p">$</span><span class="nv">_HOSTGZHOST</span><span class="p">$</span><span class="w"> </span><span class="p">$</span><span class="nv">ARG2</span><span class="p">$</span><span class="w"> </span><span class="o">-</span><span class="nv">i</span><span class="w"> </span><span class="p">$</span><span class="nv">USER5</span><span class="p">$</span><span class="o">/</span><span class="nv">smartos</span><span class="o">-</span><span class="nv">age</span><span class="o">-</span><span class="nv">check</span><span class="o">-</span><span class="nv">key</span><span class="w"> </span><span class="p">$</span><span class="nv">HOSTNAME</span><span class="p">$</span>
<span class="p">}</span>
</code></pre></div>
<p>With:</p>
<ul>
<li><code>$_HOSTGZHOST$</code> having the Global Zone host</li>
<li><code>$ARG2$</code> being a placeholder for optional parameters (such as overriding the timelines)</li>
<li><code>$USER5$</code> pointing to our directory for storing ssh keys</li>
<li><code>$HOSTNAME$</code> the name of the VM to check</li>
</ul>
<h2>Results</h2>
<p>In the end, I found a few more systems that were out of date than I was
expecting, including one I could have sworn I'd refreshed just earlier
this week. So, I'm pretty happy with the system.</p>
Subtasks and Redirection2023-05-06T07:30:00-04:002023-05-06T07:30:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-05-06:/subtasks-and-redirection.html<h2>Background</h2>
<p>As part of an ongoing effort to keep Cartographica up to date with recent
changes in libraries that we compile from source, notably
<a href="https://gdal.org">GDAL</a> and <a href="https://proj.org">Proj</a>, I'm in the midst
of a refresh of those <a href="https://www.gaige.net/git-subtrees-for-perforce-users.html">subtrees</a>
in the frameworks that I build from them. Over the past few years …</p><h2>Background</h2>
<p>As part of an ongoing effort to keep Cartographica up to date with recent
changes in libraries that we compile from source, notably
<a href="https://gdal.org">GDAL</a> and <a href="https://proj.org">Proj</a>, I'm in the midst
of a refresh of those <a href="https://www.gaige.net/git-subtrees-for-perforce-users.html">subtrees</a>
in the frameworks that I build from them. Over the past few years, both
of these projects have expanded test coverage and modernized their build
architectures (using <a href="http://cmake.org">CMake</a>) and I've improved validation
and coverage by integrating these tests into my Xcode build environment.</p>
<p>Up until the Cartographica 1.6 release, where I made available the
<a href="https://blog.cartographica.com/command-line-tools-for-gdal-and-proj.html">Command Line Tools for GDAL and PROJ</a>,
I didn't have a way to do acceptance testing on the final product,
so I integrated these tests into the unit tests for the frameworks.</p>
<h2>A Problem with Shell Redirection</h2>
<p>For many of the tests for both PROJ especially, the tests involve invoking
CLI commands with a set of parameters and validating that the exact results
are as expected. In order to support this, I created an Objective-C class
that spawns a <code>/bin/sh</code> shell (although the specific flavor doesn't seem
to have much effect on the problem) using the executable-bit-marked shell
script as <em>arg0</em> with the necessary arguments and environment variables in
place.</p>
<p>This has worked well since I build this structure in 2014. However, the most
recent updates elicited failures based on the diffs in the tests not succeeding.
First check was to run the test manually, which resulted in... succcess.
That was a bit unexpected, since I'm running the same commands in effectively
the same environment in both cases...but, of course, it is not.</p>
<p>To run the tests from within the <a href="https://developer.apple.com/documentation/xctest">XCTest</a>
structure, I am running in code, and that means that I need to spawn the
task using a sub-shell, which in my case involves spawning an
<a href="https://developer.apple.com/documentation/foundation/nstask/"><code>NSTask</code></a>, and
waiting for it to complete in order to gather the results.</p>
<p>Looking at the results, the key difference is that when run in my <code>NSTask</code>, the
redirection of the <code>stdout</code> and <code>stderr</code> to the same location <em>in the script</em>
works differently than it does from the command line. When run from the command
line, they are separately buffered, causing the results to appear as:</p>
<div class="codehilite"><pre><span></span><code>Attempt to use coordinate operation Inverse of WGS 84 to EGM2008 height (1) failed.
49 2 0 <span class="gs">* *</span> inf
</code></pre></div>
<p>When run inside of the <code>NSTask</code>, the results are a less useful:</p>
<div class="codehilite"><pre><span></span><code><span class="mf">49</span><span class="w"> </span><span class="mf">2</span><span class="w"> </span><span class="mf">0</span><span class="w"> </span><span class="n">Attempt</span><span class="w"> </span><span class="kr">to</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="n">coordinate</span><span class="w"> </span><span class="n">operation</span><span class="w"> </span><span class="n">Inverse</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">WGS</span><span class="w"> </span><span class="mf">84</span><span class="w"> </span><span class="kr">to</span><span class="w"> </span><span class="n">EGM2008</span><span class="w"> </span><span class="n">height</span><span class="w"> </span><span class="p">(</span><span class="mf">1</span><span class="p">)</span><span class="w"> </span><span class="n">failed</span><span class="mf">.</span>
<span class="o">*</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">inf</span>
</code></pre></div>
<p>The code for the underlying command echos to <code>stdout</code> the initial coordinates (<code>49 2 0 </code>)
before the error occurs, then sends the error to <code>stderr</code> and then continue to print
the result to <code>stderr</code>, including the <code>\n</code>, signalling EOL, and flushing the buffer.</p>
<p>It's not at all clear why the behavior of the buffering is working differently during the
script executed from withing the shell directly rather than from the script executed
from within the <code>NSTask</code>. In this case, the actual redirection happens as part of the
script and not part of the original shell from which the script is being run. I speculate
that there's some kind of default handlign that is getting passed through to the
script from the original shell, and when I use <code>NSTask</code> it is coming from there instead.</p>
<p>The code is pretty straigthforward:</p>
<div class="codehilite"><pre><span></span><code><span class="p">-</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="nf">runScriptTest:</span><span class="p">(</span><span class="bp">NSString</span><span class="o">*</span><span class="p">)</span><span class="nv">script</span><span class="w"> </span><span class="nf">withExecutable:</span><span class="p">(</span><span class="bp">NSString</span><span class="o">*</span><span class="p">)</span><span class="nv">executable</span><span class="w"> </span><span class="nf">andArguments:</span><span class="p">(</span><span class="bp">NSArray</span><span class="o"><</span><span class="bp">NSString</span><span class="o">*>*</span><span class="w"> </span><span class="n">_Nullable</span><span class="p">)</span><span class="nv">userArguments</span>
<span class="p">{</span>
<span class="w"> </span><span class="bp">NSBundle</span><span class="w"> </span><span class="o">*</span><span class="n">testBundle</span><span class="w"> </span><span class="o">=</span><span class="p">[</span><span class="bp">NSBundle</span><span class="w"> </span><span class="n">bundleForClass</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="nb">self</span><span class="w"> </span><span class="k">class</span><span class="p">]];</span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">executablePath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">testBundle</span><span class="w"> </span><span class="n">pathForAuxiliaryExecutable</span><span class="o">:</span><span class="w"> </span><span class="n">executable</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">executablePath</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need executable %@"</span><span class="p">,</span><span class="w"> </span><span class="n">executable</span><span class="p">);</span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">scriptPath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">testBundle</span><span class="w"> </span><span class="n">pathForResource</span><span class="o">:</span><span class="w"> </span><span class="n">script</span><span class="w"> </span><span class="n">ofType</span><span class="o">:</span><span class="nb">nil</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">scriptPath</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need script %@"</span><span class="p">,</span><span class="w"> </span><span class="n">script</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">runDir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">NSProcessInfo</span><span class="p">.</span><span class="n">processInfo</span><span class="p">.</span><span class="n">environment</span><span class="p">[</span><span class="s">@"TMPDIR"</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">runDir</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need runPath %@"</span><span class="p">,</span><span class="w"> </span><span class="n">script</span><span class="p">);</span>
<span class="w"> </span><span class="n">XCTAssertNotEqualObjects</span><span class="p">(</span><span class="n">runDir</span><span class="p">,</span><span class="w"> </span><span class="s">@"/"</span><span class="p">);</span>
<span class="w"> </span><span class="n">NSTask</span><span class="w"> </span><span class="o">*</span><span class="n">childTask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[[</span><span class="n">NSTask</span><span class="w"> </span><span class="n">alloc</span><span class="p">]</span><span class="w"> </span><span class="n">init</span><span class="p">];</span>
<span class="w"> </span>
<span class="w"> </span><span class="bp">NSArray</span><span class="w"> </span><span class="o">*</span><span class="n">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="l">@[</span><span class="n">scriptPath</span><span class="p">,</span><span class="w"> </span><span class="n">executablePath</span><span class="p">,</span><span class="w"> </span><span class="nb">self</span><span class="p">.</span><span class="n">nadPath</span><span class="l">]</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">userArguments</span><span class="p">)</span>
<span class="w"> </span><span class="n">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">arguments</span><span class="w"> </span><span class="n">arrayByAddingObjectsFromArray</span><span class="o">:</span><span class="w"> </span><span class="n">userArguments</span><span class="p">];</span>
<span class="w"> </span><span class="n">childTask</span><span class="p">.</span><span class="n">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">arguments</span><span class="p">;</span>
<span class="w"> </span><span class="n">childTask</span><span class="p">.</span><span class="n">executableURL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="bp">NSURL</span><span class="w"> </span><span class="n">fileURLWithPath</span><span class="o">:</span><span class="w"> </span><span class="s">@"/bin/sh"</span><span class="p">];</span>
<span class="w"> </span><span class="n">childTask</span><span class="p">.</span><span class="n">currentDirectoryURL</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="bp">NSURL</span><span class="w"> </span><span class="n">fileURLWithPath</span><span class="o">:</span><span class="w"> </span><span class="n">runDir</span><span class="p">];</span>
<span class="w"> </span><span class="n">childTask</span><span class="p">.</span><span class="n">environment</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="nb">self</span><span class="w"> </span><span class="n">environmentWithResources</span><span class="p">];</span>
<span class="w"> </span>
<span class="w"> </span><span class="bp">NSError</span><span class="w"> </span><span class="o">*</span><span class="n">error</span><span class="p">;</span>
<span class="w"> </span><span class="n">XCTAssertTrue</span><span class="p">([</span><span class="n">childTask</span><span class="w"> </span><span class="n">launchAndReturnError</span><span class="o">:</span><span class="w"> </span><span class="o">&</span><span class="n">error</span><span class="p">],</span><span class="w"> </span><span class="s">@"Launch failed %@"</span><span class="p">,</span><span class="w"> </span><span class="n">error</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="p">[</span><span class="n">childTask</span><span class="w"> </span><span class="n">waitUntilExit</span><span class="p">];</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">childTask</span><span class="w"> </span><span class="n">terminationStatus</span><span class="p">];</span>
<span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">status</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>A minimal C program doesn't have any problem with this:</p>
<div class="codehilite"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><unistd.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><string.h></span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">**</span><span class="n">argv</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">env</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s">"PROJ_DATA=Proj4Tests.xctest/Contents/Resources/for_tests"</span><span class="p">,</span>
<span class="w"> </span><span class="nb">NULL</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">result</span><span class="p">;</span>
<span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">execle</span><span class="p">(</span><span class="w"> </span><span class="s">"/bin/sh"</span><span class="p">,</span><span class="w"> </span><span class="s">"/bin/sh"</span><span class="p">,</span><span class="w"> </span><span class="s">"Proj4Tests.xctest/Contents/Resources/testvarious"</span><span class="p">,</span><span class="w"> </span><span class="s">"Proj4Tests.xctest/Contents/MacOS/cs2cs"</span><span class="p">,</span><span class="w"> </span><span class="s">"Proj4Tests.xctest/Contents/Resources/for_tests"</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="n">env</span><span class="p">);</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"result = %d (%s)"</span><span class="p">,</span><span class="w"> </span><span class="n">result</span><span class="p">,</span><span class="w"> </span><span class="n">strerror</span><span class="p">(</span><span class="n">result</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div>
<h2>A solution by replacing NSTask</h2>
<p>Doing some further experimentation, I don't end up with interleaved output
on the subprocess is I use <code>posix_spawn</code> instead of spawning with <code>NSTask</code>.</p>
<p>Adapting my original code, this seems to work:</p>
<div class="codehilite"><pre><span></span><code><span class="p">-</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="nf">runScriptTest:</span><span class="p">(</span><span class="bp">NSString</span><span class="o">*</span><span class="p">)</span><span class="nv">script</span><span class="w"> </span><span class="nf">withExecutable:</span><span class="p">(</span><span class="bp">NSString</span><span class="o">*</span><span class="p">)</span><span class="nv">executable</span><span class="w"> </span><span class="nf">andArguments:</span><span class="p">(</span><span class="bp">NSArray</span><span class="o"><</span><span class="bp">NSString</span><span class="o">*>*</span><span class="w"> </span><span class="n">_Nullable</span><span class="p">)</span><span class="nv">userArguments</span>
<span class="p">{</span>
<span class="w"> </span><span class="bp">NSBundle</span><span class="w"> </span><span class="o">*</span><span class="n">testBundle</span><span class="w"> </span><span class="o">=</span><span class="p">[</span><span class="bp">NSBundle</span><span class="w"> </span><span class="n">bundleForClass</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="nb">self</span><span class="w"> </span><span class="k">class</span><span class="p">]];</span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">executablePath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">testBundle</span><span class="w"> </span><span class="n">pathForAuxiliaryExecutable</span><span class="o">:</span><span class="w"> </span><span class="n">executable</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">executablePath</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need executable %@"</span><span class="p">,</span><span class="w"> </span><span class="n">executable</span><span class="p">);</span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">scriptPath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">testBundle</span><span class="w"> </span><span class="n">pathForResource</span><span class="o">:</span><span class="w"> </span><span class="n">script</span><span class="w"> </span><span class="n">ofType</span><span class="o">:</span><span class="nb">nil</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">scriptPath</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need script %@"</span><span class="p">,</span><span class="w"> </span><span class="n">script</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">runDir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">NSProcessInfo</span><span class="p">.</span><span class="n">processInfo</span><span class="p">.</span><span class="n">environment</span><span class="p">[</span><span class="s">@"TMPDIR"</span><span class="p">];</span>
<span class="w"> </span><span class="n">XCTAssertNotNil</span><span class="p">(</span><span class="w"> </span><span class="n">runDir</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need runPath %@"</span><span class="p">,</span><span class="w"> </span><span class="n">script</span><span class="p">);</span>
<span class="w"> </span><span class="n">XCTAssertNotEqualObjects</span><span class="p">(</span><span class="n">runDir</span><span class="p">,</span><span class="w"> </span><span class="s">@"/"</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="bp">NSArray</span><span class="w"> </span><span class="o">*</span><span class="n">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="l">@[</span><span class="n">scriptPath</span><span class="p">,</span><span class="w"> </span><span class="n">executablePath</span><span class="p">,</span><span class="w"> </span><span class="nb">self</span><span class="p">.</span><span class="n">nadPath</span><span class="l">]</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">userArguments</span><span class="p">)</span>
<span class="w"> </span><span class="n">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">arguments</span><span class="w"> </span><span class="n">arrayByAddingObjectsFromArray</span><span class="o">:</span><span class="w"> </span><span class="n">userArguments</span><span class="p">];</span>
<span class="w"> </span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">env</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">[</span><span class="bp">NSString</span><span class="w"> </span><span class="n">stringWithFormat</span><span class="o">:</span><span class="w"> </span><span class="s">@"PROJ_DATA=%@"</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="nb">self</span><span class="p">.</span><span class="n">nadPath</span><span class="w"> </span><span class="n">stringByAppendingPathComponent</span><span class="o">:</span><span class="s">@"for_tests"</span><span class="p">]].</span><span class="n">UTF8String</span><span class="p">,</span>
<span class="w"> </span><span class="nb">NULL</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">args</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s">"/bin/sh"</span><span class="p">,</span>
<span class="w"> </span><span class="n">scriptPath</span><span class="p">.</span><span class="n">UTF8String</span><span class="p">,</span>
<span class="w"> </span><span class="n">executablePath</span><span class="p">.</span><span class="n">UTF8String</span><span class="p">,</span>
<span class="w"> </span><span class="nb">self</span><span class="p">.</span><span class="n">nadPath</span><span class="p">.</span><span class="n">UTF8String</span><span class="p">,</span>
<span class="w"> </span><span class="nb">NULL</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">pthread_chdir_np</span><span class="p">(</span><span class="n">runDir</span><span class="p">.</span><span class="n">UTF8String</span><span class="p">);</span>
<span class="w"> </span>
<span class="w"> </span><span class="kt">pid_t</span><span class="w"> </span><span class="n">pid</span><span class="p">;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">success</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">posix_spawn</span><span class="p">(</span><span class="w"> </span><span class="o">&</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="s">"/bin/sh"</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="k">const</span><span class="o">*</span><span class="p">)</span><span class="n">args</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="k">const</span><span class="o">*</span><span class="p">)</span><span class="n">env</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">success</span><span class="o">==</span><span class="mi">-1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Fork failed"</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="p">(</span><span class="mi">-1</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"parent sees child %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">pid</span><span class="p">);</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">status</span><span class="p">;</span>
<span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">waitpid</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">status</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pid</span><span class="o"><</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">errno</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">EINTR</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">waitpid</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">status</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">pid</span><span class="o"><</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"Error waiting %d %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">pid</span><span class="p">,</span><span class="w"> </span><span class="n">errno</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="p">(</span><span class="mi">-2</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">"child completed with %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="n">status</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">status</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Two things of note here:</p>
<ul>
<li><code>pthread_chdir_np</code> is not a public method for macOS -- other third party
applications, like Chrome use it, but it's not sanctioned and could
go away. I'm less concerned about this in a test jig than in code that
would go to end users.</li>
<li>The little dance around <code>waitpid</code> being called twice is related to
receiving a signal, which I am pretty certain is <code>SIGCHLD</code> being sent.
However, I'm not comfortable ignoring it because I may not be the only
one spawning a child task.</li>
</ul>
fp-concat Accuracy2023-05-06T05:53:00-04:002023-05-06T05:53:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-05-06:/fp-concat-accuracy.html<p>My previous post about <a href="https://www.gaige.net/proj-floating-point-error-investigation.html">proj floating point investigation</a>
discussed an issue that I'd tracked down to the OS level. However, it's clear
that this relates to an underlying change to code compiled by Xcode (and/or
the <a href="https://llvm.org">LLVM toolchain</a> that it is built upon).</p>
<p>Based on a <a href="https://mjtsai.com/blog/2023/04/03/xcode-14-3/">post about Xcode …</a></p><p>My previous post about <a href="https://www.gaige.net/proj-floating-point-error-investigation.html">proj floating point investigation</a>
discussed an issue that I'd tracked down to the OS level. However, it's clear
that this relates to an underlying change to code compiled by Xcode (and/or
the <a href="https://llvm.org">LLVM toolchain</a> that it is built upon).</p>
<p>Based on a <a href="https://mjtsai.com/blog/2023/04/03/xcode-14-3/">post about Xcode 14.3</a>
in <a href="https://mjtsai.com/blog/">Michael Tsai's blog</a>, I started looking at a
change in the compiler around the handling of <code>fp-contract</code>, which controls
the use of contractions in optimizing floating point operations in the compiler.</p>
<p>There's certainly beeen some <a href="https://discourse.llvm.org/t/fp-contraction-fma-on-by-default/64975">debate about the change</a>
that was introduced in Clang 14 which changed the handling of the
<code>fp-contract</code> flag, and I'm not going to take a stand on which is "better",
but I am going to note that the change was unexpected (in a minor release)
and had notable, if not significant effects on floating point handling in
Cartographica.</p>
<p>The change in behavior results in Xcode 14.3 (Clang 14) by default choosing
to contract Multiply and Add instructions in floating point using a
Fused Multiply Add instruction that is intended to capture rounding
betwen the operations. Although this likely makes the calculations
more accurate, it runs the risk of diverging from existing resutls
and can create compolexitiies in testing.</p>
<p>In practice, I haven't found a large number of differences, but in some
cases, there are variances that are causing some difficulties in test
management.</p>
<p>The change itself was in how the <code>fp-contract</code> (floating-point contraction)
flag was being handled by Clang to bring it more into alignment with the
standards for C/C++. The details on the flag handling are a bit esoteric
so I'll leave that for the reader, but there are a
<a href="https://clang.llvm.org/docs/UsersManual.html#controlling-floating-point-behavior">variety of options</a>
for the setting and the change was in the default handling between Clang 13
and Clang 14, effectively causing it to move from <code>off</code> to <code>on</code> by default.</p>
<p>If you want to see an illustration of what this does from a code generation
perspective, there's a good <a href="https://godbolt.org/z/WK8zMzq8s">comparison using godbolt</a>
between default Clang 13 and Clang 14, as well as Clang 14 with
<code>-ffp-contract=off</code>, showing the behavior change.
(
<a href="https://godbolt.org/#z:OYLghAFBqd5TKALEBjA9gEwKYFFMCWALugE4A0BIEAZgQDbYB2AhgLbYgDkAjF%2BTXRMiAZVQtGIHgBYBQogFUAztgAKAD24AGfgCsp5eiyahUAUgBMAIUtXyKxqiIEh1ZpgDC6egFc2TEABmAA5ydwAZAiZsADk/ACNsUikAdnIAB3QlYhcmL19/INDM7OchSOi4tkTknjSHbCdckSIWUiJ8vwCQ%2B2xHMqYWtqIK2ISk1PtW9s7CnqVpkaix6om6gEp7dB9SVE4uGnp0FiIAahV0tpPsWiOT05ZyU8Pjs/j107MUmy0AQVOAQ8AFRmQIAEXioJ%2B/0BLFsoLBgShZj%2BgNOpGwRB2TAeyL%2BXzBKN%2BRJe92yJkYAH0FtcOMJbq8Hk9SW8Pl9oWiMVjSDiWED4p9rKckYFoQSiQB6CWnYCoVAgZ7pAC0GGEpBYTgRQi1NBon3BpyY6FOqqI6qaQgAdJLpWjAQBxDweU6YdDYJRMMBcM4AayNAHdTkh0IGSKd6AQ2MQTfJzQNTmGotkcOdWkRsHSiEpzsbLBZtRYLKc2D4Fi6CEoWPFGKdiDaZXKFTgaCwfPQiAiW2WJP6WABPbOmuNEkd/KUmowmBU0ZVDjUd8HoXWAisJpDYF3YFttjtj6WoSegRUq2PzrU4gEKudOU7%2B4hIKIPVO05i737jg/GI8zk9qs/grszgVCMfQ3R0PC4TZ6G4ABWfgAi4HRyHQbgPHhIUlG2XYN0sQI%2BHIIhtEgzYfRAaRpEtCwAE5C2CGCeC0QJGOkFI0mgrhpH4NgQBgrRyAQpCUK4fglBAPjCMQyDyDgWAUAwNh0gYJJKGoeTFMYZJPxMBitD4nAADcCD2AA1AhsH9AB5dJmG4fC6HbJJRIgeIiPIeIojaPtbP4dzWFIPsLPiXRGgk/D5MzCymHoLzJPIHA2C/SRYsIDEmn091XOwdRGh8dNXKidN2KQiN4nVfyvBwbyCNISNvM2Q4WGAJRTPMqybN4fhBGEMQJE4GQ5GEZQ1E0WL9B4QwvzQdC7BK0TIE2dB0gGUSuCVdRUFOJULMCTaEt2JAESUH0%2ByMUCRL6ELcjcJhPG8LoDAiZYqhqAwShyIRZgCca3oGUZnrWXp%2BmaRZPoMBoLUGRY/vGWopmGUHxppdpodWWpNkwnY9ikKDYPg1yhNOdRggANiVYnpAnL9TgYy0tFp04IHwYgyEFPD1n4CSdHWTZ1xYHBkggHGOK4ni%2BIE/ghJEsSCKI7nyFImCKOJiwQmCHhlekYILC0YJQnYwI8diyWZckzYZIQeAIDk9AFKUigqAgNS7ZAYAaLi7BDJMszLOshC7IYdNSCclzYt8zyqrD/zAuCpwqvC19Iui1z4sS/YkJSy70pWpCspyvLYoKvpXJKsq%2Bwq/Z8LNWqOvqowmpan32rs%2BQeskfqusUFQNFc/QLAmqdzGsWxDAIeI5sF5CltyFa1o2radqVPbUAO8EjpOlgzsBy7XAgdwEbCG6UZe76snevI7sKE/SlyI%2BAfBgYhhmC%2Bvq3iHH6WSoYbBkHn%2B/4Zb7RlsTGfUhZwX4vjbghMSZkwprKDaVFKIMyZiQUgrMeDsxNlzHm2A%2BYTAnuxTi5BuK8XAUbbgUtxKy2kogK2aAbbqWUg7J2Gk0CHhkLpd2ntsANzan7TqAdHLUBDkhSOMV8KiOjiFOONsIpRRiunDMqdk4EFSs4LOmVsqoFymnfghcir8BLp5cuVUq7cRrgIOuzVva8Kqh3VufVZAdyGt3UaQR%2B6mGmiPMe8AFpTyEDPdam1tq7TaMvQ6x1TrYHOkDHee9f7jUep/VGr1T4DH3j9G%2BT0v6Iwum/H%2BBQX732Bv/LJyTEb5PuuUkpSTj7oywljdBhhcakMEpAompNyaUxMNTCidMtBIMICgtBGDObEXILzfm1ASJkRgpaQIKRiY8B4MEeZ9EUhUR4BYGCTThZENFi0iW5D7DS1GXLUi0hZnzMWcs1ZdQNlbJ2QbA5yEjmnKFhYQ2rThKYLGelIOV1pBAA%3D%3D">longer godbolt link here if the short one ever goes stale</a>).</p>
<p>I'm still on the fence over whether this is actually a code problem or a test
problem, but at the moment, it's really feeling more like the latter.
The IEEE Floating Point standards define the fused-multiply-add operator and it's
clearly intended to remove some error in the combining of floating point
operations while also improving speed.</p>
xcodes for xcode switching2023-04-29T15:50:00-04:002023-04-29T15:50:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-29:/xcodes-for-xcode-switching.html<p>As part of digging through my various problems with Xcode 14.3 (Feedback
FB12154691, FB12154887, and some
<a href="https://www.gaige.net/proj-floating-point-error-investigation.html">test case issues involving floating point math</a>),
I needed to install Xcode 14.2 to move my buildfarm backwards.
Although this didn't enitrely fix the problem, it was an essential
element of the …</p><p>As part of digging through my various problems with Xcode 14.3 (Feedback
FB12154691, FB12154887, and some
<a href="https://www.gaige.net/proj-floating-point-error-investigation.html">test case issues involving floating point math</a>),
I needed to install Xcode 14.2 to move my buildfarm backwards.
Although this didn't enitrely fix the problem, it was an essential
element of the debugging and remediation.</p>
<p>For manual work, there's a great
<a href="https://xcodereleases.com">list of Xcode releases</a> available with
direct links to Apple's downloads and release notes.</p>
<p>Since I was in the need to do this across five different machines,
automation was on my mind, so I looked for the latest in tooling to
help with this install process.</p>
<p>The latest in this field is <a href="https://github.com/XcodesOrg/xcodes"><code>xcodes</code></a>,
open source tooling for installing one or more copies of xcode and
switching between them.</p>
<h2><code>xcodes</code> Commands</h2>
<ul>
<li>Use <code>xcodes installed</code> to find out which versions are installed</li>
<li>Use <code>xcodes install XX.YY</code> to install a specific version</li>
<li>Use <code>xcodes select XX.YY</code> to make the specified version the default</li>
<li>Use <code>xcodes uninstall XX.YY</code> to uninstall a specific version</li>
</ul>
<h2>Minimizing traffic and logins</h2>
<p>To retrieve the xcode installer from Apple, you need to be logged in
to a developer account, and that means credentials. Coordinating that
with automation is painful and also would result in pulling every
installer I need (each time I need it) across the internet.</p>
<p>As I'll be automating installation on multiple systems, I decided that
I'd cache the items that I need in order to save time and bandwidth.</p>
<p>To download the packages into your cache:</p>
<pre><code>xcodes download --directory CACHE_DIR 13.2.1
</code></pre>
<h2>Automating installation</h2>
<p>For installation (via Ansible), I'm using the following
(assuming <code>item.version</code> contains the version on input):</p>
<div class="codehilite"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">copy xcode installer</span>
<span class="w"> </span><span class="nt">copy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">xcode_cache</span><span class="nv"> </span><span class="s">}}/{{</span><span class="nv"> </span><span class="s">item.package</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">root_home</span><span class="nv"> </span><span class="s">}}/Downloads/{{</span><span class="nv"> </span><span class="s">item.package</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">owner</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">group</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s">'0644'</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s">"install</span><span class="nv"> </span><span class="s">xcode</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item.version</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span>
<span class="w"> </span><span class="nt">cmd</span><span class="p">:</span><span class="w"> </span><span class="s">"xcodes</span><span class="nv"> </span><span class="s">install</span><span class="nv"> </span><span class="s">--experimental-unxip</span><span class="nv"> </span><span class="s">--path</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">root_home</span><span class="nv"> </span><span class="s">}}/Downloads/{{</span><span class="nv"> </span><span class="s">item.package</span><span class="nv"> </span><span class="s">}}</span><span class="nv"> </span><span class="s">{{</span><span class="nv"> </span><span class="s">item.version</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">become</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">remove installer</span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">root_home</span><span class="nv"> </span><span class="s">}}/.Trash/{{</span><span class="nv"> </span><span class="s">item.package</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">absent</span>
<span class="w"> </span><span class="nt">ignore_errors</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</code></pre></div>
<p>This uses <code>xcodes</code> to install without downloading, based on the <code>xip</code> file
(along with enabling the experimental fast unxip code).</p>
<p>This code is called using <code>include_tasks</code> from a loop in my main
ansible ci-bot file that installs appropriate versions:</p>
<div class="codehilite"><pre><span></span><code><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">determine current xcodes</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">shell</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="nt">cmd</span><span class="p">:</span><span class="w"> </span><span class="s">"xcodes</span><span class="nv"> </span><span class="s">installed</span><span class="nv"> </span><span class="s">|</span><span class="nv"> </span><span class="s">cut</span><span class="nv"> </span><span class="s">-f1</span><span class="nv"> </span><span class="s">-d'</span><span class="nv"> </span><span class="s">'"</span>
<span class="w"> </span><span class="w w-Error"> </span><span class="nt">register</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">xcodes_installed</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">install xcode if missing</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">include_tasks</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ci-xcode.yml</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">loop</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">xcode_versions</span><span class="nv"> </span><span class="s">}}"</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">item.version not in xcodes_installed.stdout_lines</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ansible_distribution_version >= item.min_os</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">item.max_os is not defined or (ansible_distribution_version < item.max_os )</span>
</code></pre></div>
<p>The first task gets a list of current xcodes (to keep from
reinstalling) and then installs only if it's not already
installed and the version is appropriate for the version
of macOS that we're installing on.</p>
<p><code>xcode_versions</code> looks like this:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">xcode_versions</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'13.2.1'</span>
<span class="w"> </span><span class="nt">package</span><span class="p">:</span><span class="w"> </span><span class="s">'Xcode-13.2.1+13C100.xip'</span>
<span class="w"> </span><span class="nt">min_os</span><span class="p">:</span><span class="w"> </span><span class="s">'11.3.0'</span>
<span class="w"> </span><span class="nt">max_os</span><span class="p">:</span><span class="w"> </span><span class="s">'12.0.0'</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'14.3'</span>
<span class="w"> </span><span class="nt">package</span><span class="p">:</span><span class="w"> </span><span class="s">'Xcode-14.3.0+14E222b.xip'</span>
<span class="w"> </span><span class="nt">min_os</span><span class="p">:</span><span class="w"> </span><span class="s">'13.0.0'</span>
<span class="c1"># intentionally out-of-order because 14.2 is preferred right now</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">'14.2'</span>
<span class="w"> </span><span class="nt">package</span><span class="p">:</span><span class="w"> </span><span class="s">'Xcode-14.2.0+14C18.xip'</span>
<span class="w"> </span><span class="nt">min_os</span><span class="p">:</span><span class="w"> </span><span class="s">'12.5.0'</span>
</code></pre></div>
Proj Floating Point Error Investigation2023-04-29T09:27:00-04:002023-04-29T09:27:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-29:/proj-floating-point-error-investigation.html<h2>TL;DR</h2>
<p>MacOS 13.3 or 13.3.1 incorporated a change that is affecting calculations in
proj for applications running on those versions of the OS. The change appears
to be relatviely subtle, only affecting a single test in a single projection
and only on x86_64, not arm, but …</p><h2>TL;DR</h2>
<p>MacOS 13.3 or 13.3.1 incorporated a change that is affecting calculations in
proj for applications running on those versions of the OS. The change appears
to be relatviely subtle, only affecting a single test in a single projection
and only on x86_64, not arm, but nonetheless resulting in a failure on macOS
in the standard gie tests for <a href="https://github.com/OSGeo/PROJ">proj</a>.</p>
<p>Since the tests work successfully on other platforms and on versions of
macOS prior to 13.3, I'm going to assume for now that it is a variation
from the floating point execution that is peculiar to macOS and only on
Intel CPUs. I did file an <a href="https://github.com/OSGeo/PROJ/issues/3714">issue</a>
with the project just in case we want to believe this isn't a bug in macOS.</p>
<h2>Investigating</h2>
<p>Last weekend, I decided to finally upgrade my macOS build farm, which I'd
been putting off doing because it would require 5 macOS upgrades and
5 Xcode upgrades. At this point, my automation for the former is inadequate
so I usually do that process by hand. The latter, I had similarly done
by hand (until moving that work to <code>xcodes</code> this week).</p>
<p>Unfortunately, I hadn't upgraded my desktop MacPro to 13.3.1 prior to
doing the upgrades in the build farm, so I didn't know what kind of fun
I had in store.</p>
<p>Once I'd finished the upgrades, I ran a test build across the build farm
and the Intel-based Ventura (13.3.1) machine was the only one that was
failing. I went back to my MacPro (still running 13.2.1) and I had no
problems. A few days later, I upgraded the MacPro to 13.3.1, and
suddenly, it was causing the same problem. At this point, I've isolated
the problem (somewhat accidentally) to the Intel macOS 13.3.1 plafforms.</p>
<p>As a side note, testing in this case was made much easier by the fact
that I'd released the proj and gdal
<a href="https://blog.cartographica.com/command-line-tools-for-gdal-and-proj.html">CLI tools</a>
with the last major release of Cartographica, meaning I could run the
functional portion of the tests without having to use xctest.</p>
<p>I was having some other problems with Xcode 14.3, so I decided to
back down the Xcode version thinking that may be causing my problem.
(some more on that is detailed in
<a href="https://www.gaige.net/xcodes-for-xcode-switching.html">Xcodes for Xcode Switching</a>).</p>
<p>After downgrading to Xcode 14.2, the aforementioned inaccuracy was
still happening, which seemed to rule out Xcode as the cause of
<em>this</em> problem.</p>
<p>Unfortunately, I'd already upgraded all of my macOS 13.2 machines to
13.3, so I decided to create a VM to run the regression tests against
macOS 13.2. Thanks to the <a href="https://mrmacintosh.com/macos-ventura-13-full-installer-database-download-directly-from-apple/">list of Apple macOS Downloads for Ventura</a>,
I was able to download the installer and create a VM under
<a href="https://www.parallels.com">Parallels</a> to do my tests. It took a bit
of time, but once operating, I could confirm the problems were only
happening on macOS 13.3.1 on Intel-based CPUs.</p>
<p>I'll post an end to this story when it happens, but it's a bit frightening
to see floating point changes coming in without significant notification
in minor macOS updates.</p>
<p>I did find 2 potential work-arounds, which I detailed in my Issue with
the Proj owners above. I'll post a follow-up article when the final
issue is dispositioned.</p>
Renovating GitLab registries2023-04-23T09:35:00-04:002023-04-23T09:35:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-23:/renovating-gitlab-registries.html<p>I've already written a bit about using renovate to keep dependencies current using
Renovate On Prem in
<a href="https://www.gaige.net/renovating-gitlab-repos.html">Renovating GitLab Repos</a>.
This has been working well. However, there are a couple of twists
that I figured I'd document in the event that people run into them.</p>
<p>For single-repositories with public dependencies …</p><p>I've already written a bit about using renovate to keep dependencies current using
Renovate On Prem in
<a href="https://www.gaige.net/renovating-gitlab-repos.html">Renovating GitLab Repos</a>.
This has been working well. However, there are a couple of twists
that I figured I'd document in the event that people run into them.</p>
<p>For single-repositories with public dependencies, the default configuration works without
much tweaking. As I mentioned in my previous article, there are a few nuances for dealing
with git submodules and other dependency types that are served by gitlab.</p>
<p>I noticed this first with the <code>git-submodules</code> module, basically that it wasn't
authenticating and thus wasn't able to determine updates for self-hosted submodules.
Additionally, as I expanded use to other repositories, I noticed that checking
gitlab-hosted helm charts (<code>helm</code> module) and gitlab-hosted docker containers
(<code>docker</code> module) were also failing. In these cases, it is unclear (even with
debugging on) whether the token auth was being used due to the prior <code>hostMatch</code>
records or not. However, I was able to confirm that for the docker registries, at least
I couldn't log in with a bearer token, and I'm assuming a similar problem was at play
with the helm repository.</p>
<p>The fix in my configuration was a <code>hostRules</code> array with a
set of <code>hostMatch</code> directives which are used to
map the authentication mechanisms to specific hosts.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">"hostRules"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"matchHost"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{ requiredEnv "</span><span class="err">CI_SERVER_HOST</span><span class="s2">" }}"</span><span class="p">,</span><span class="w"> </span><span class="nt">"token"</span><span class="p">:</span><span class="s2">"{{ requiredEnv "</span><span class="err">RENOVATE_GITLAB_TOKEN</span><span class="s2">" }}"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"matchHost"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{ requiredEnv "</span><span class="err">CI_SERVER_HOST</span><span class="s2">" }}"</span><span class="p">,</span><span class="w"> </span><span class="nt">"hostType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"docker"</span><span class="p">,</span><span class="w"> </span><span class="nt">"username"</span><span class="p">:</span><span class="w"> </span><span class="s2">"token"</span><span class="p">,</span><span class="w"> </span><span class="nt">"password"</span><span class="p">:</span><span class="s2">"{{ requiredEnv "</span><span class="err">RENOVATE_GITLAB_TOKEN</span><span class="s2">" }}"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"matchHost"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{ requiredEnv "</span><span class="err">CI_SERVER_HOST</span><span class="s2">" }}"</span><span class="p">,</span><span class="w"> </span><span class="nt">"hostType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"helm"</span><span class="p">,</span><span class="w"> </span><span class="nt">"username"</span><span class="p">:</span><span class="w"> </span><span class="s2">"token"</span><span class="p">,</span><span class="w"> </span><span class="nt">"password"</span><span class="p">:</span><span class="s2">"{{ requiredEnv "</span><span class="err">RENOVATE_GITLAB_TOKEN</span><span class="s2">" }}"</span><span class="w"> </span><span class="p">}</span>
<span class="p">]</span>
</code></pre></div>
<p>Originally, I'd expected that Renovate would create a default <code>hostRule</code>
based on the server and gitlab token. However, even if that is
the case for some items, it doesn't work for all of them.
I've reported this as a shortcoming, as I would
expect that to try the current token (basically what I'm forcing to happen here), but
it does not.</p>
<p>These three lines effectively match the <code>CI_SERVER_HOST</code> (the
gitlab server) for authentication by default to the <code>RENOVATE_GITLAB_TOKEN</code>
using a bearer token (hence the use of <code>token</code>) and then override
that for both the <code>docker</code> and <code>helm</code> repositories because they
require <code>username</code> and <code>password</code>.</p>
<p><strong>Warning</strong> this does store the token in a clear text configuration file
instead of using Kubernetes Secrets.</p>
Renovating Ansible2023-04-17T06:20:00-04:002023-04-17T06:20:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-17:/renovating-ansible.html<p>Most of the system administration work that I do has been automated using
Ansible, as I've mentioned in posts here, including
<a href="https://www.gaige.net/deploying-with-gitlab.html">Deploying with GitLab</a>.</p>
<p>Now that I've got Renovate in place
(<a href="https://www.gaige.net/renovating-gitlab-repos.html">Renovating GitLab Repos</a>),
I am starting to look at how to expand beyond my existing automations
in order to …</p><p>Most of the system administration work that I do has been automated using
Ansible, as I've mentioned in posts here, including
<a href="https://www.gaige.net/deploying-with-gitlab.html">Deploying with GitLab</a>.</p>
<p>Now that I've got Renovate in place
(<a href="https://www.gaige.net/renovating-gitlab-repos.html">Renovating GitLab Repos</a>),
I am starting to look at how to expand beyond my existing automations
in order to let the computers do a bit more of the work.</p>
<p>This weekend's project involved experimenting with the ad hoc, regex-based
integrations with Renovate to enable renovating files that might not otherwise
be in a form that most dependency managers would recognize.</p>
<p>Conceptually, it makes a lot of sense. Renovate separates the ability to understand
different <em>datasources</em>, which provide data on new dependencies from <em>managers</em>,
which are used to determine which dependencies are used. This separation, and the
level of control and customization enabled by the configuration files, enables some
interesting use cases.</p>
<p>Thanks to documentation for
<a href="https://docs.renovatebot.com/modules/manager/regex/">Custom Manager Support using Regex</a>,
the implementation for my case was pretty straightforward.</p>
<h1>Example ansible build process (SmartOS GitLab Runner)</h1>
<p>Because Rob and I are using SmartOS, there are frequently components that need to be built
separately for the OS, because they aren't available in the package manager or are not
directly supported by vendors. One such example is the GitLab runner for SmartOS.</p>
<p>Soon after starting to use GitLab, I submitted a
<a href="https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3053">MR</a> to fix some
incompatibilities with SmartOS. That was not accepted, but was expanded upon in
the <a href="https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3189">expanded MR</a>
fixing not only SmartOS, but a number of other Unix-style OSes.</p>
<p>Thankfully, the changes have remained stable and I've had no problem pulling them forward
with each release. However, since it's not a supported OS, there's no build for it.
As with many tools, I've been building this using a bespoke Ansible script in order to make
sure that I have the latest tools and environment.</p>
<p>In my playbook, I define the build version of GitLab runner using a variable, defined as:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">vars</span><span class="p">:</span>
<span class="w"> </span><span class="nt">gitlab_runner_version</span><span class="p">:</span><span class="w"> </span><span class="s">'v15.10.1'</span>
</code></pre></div>
<p>Using the RegEx Manager, I was able to mark this dependency using a comment:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">vars</span><span class="p">:</span>
<span class="w"> </span><span class="c1"># renovate: datasource=gitlab-tags depName=gitlab-org/gitlab-runner</span>
<span class="w"> </span><span class="nt">gitlab_runner_version</span><span class="p">:</span><span class="w"> </span><span class="s">'v15.10.1'</span>
</code></pre></div>
<p>by using a custom <code>renovate.json</code> file in the project:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"regexManagers"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"fileMatch"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"^*\\.yml$"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"matchStrings"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"renovate: datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\s.*_version: '(?<currentValue>.*)'\\s"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"versioningTemplate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<p>This configuration pulls out the comment (starting with the <code>renovate:</code> string) and
then pulls the <code>datasource</code>, dependency name (<code>depName</code>), the current value of
the version (<code>currentValue</code>), and optionally the version methodology (<code>versioning</code>).</p>
<p>Once checked in, this is now recognized by Renovate and it now generates MRs as necessary:</p>
<p><img src="https://www.gaige.net/images/renovating-gitlab-runner.png" alt="MR for GitLab Runner" /></p>
<p>The MR is most of the battle, as I've got plenty of experience automating ansible through
GitLab.</p>
<p>Note: Yes, I realize that I'm not using the SmartOS runner to build the SmartOS runner
directly, and I may do that in the future. That's a project for another day, since it
requires refactoring further how I'm managing builds.</p>
Resurrecting old posts2023-04-12T04:28:00-04:002023-04-12T04:28:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-12:/resurrecting-old-posts.html<p>Seemingly appropriate for the week after Easter, I've gone through some old draft posts
and decided to publish them.</p>
<p>A couple that were mostly ready and I decided to push out:</p>
<ul>
<li><a href="https://www.gaige.net/slathering-xcode-variants.html">Slathering Xcode Variants</a></li>
<li><a href="https://www.gaige.net/test-without-building-and-spm.html">Test without building and SPM</a></li>
</ul>
<p>And one that I did a bunch of additional work and …</p><p>Seemingly appropriate for the week after Easter, I've gone through some old draft posts
and decided to publish them.</p>
<p>A couple that were mostly ready and I decided to push out:</p>
<ul>
<li><a href="https://www.gaige.net/slathering-xcode-variants.html">Slathering Xcode Variants</a></li>
<li><a href="https://www.gaige.net/test-without-building-and-spm.html">Test without building and SPM</a></li>
</ul>
<p>And one that I did a bunch of additional work and brought it up to date with this week:</p>
<ul>
<li><a href="https://www.gaige.net/moving-selenium-tests-in-house.html">Moving Selenium tests in-house</a></li>
</ul>
Moving Selenium tests in-house2023-04-11T07:45:00-04:002023-04-11T07:45:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-11:/moving-selenium-tests-in-house.html<p>Ed. Note: I started this article nearly a year ago, but got stuck on the Kubernetes piece.
Now that I've resolved that, I'm publishing it.</p>
<p>I've been a very happy user of <a href="https://saucelabs.com">SauceLabs</a>
for testing for many years. However, I don't
make a lot of use of it, and recently …</p><p>Ed. Note: I started this article nearly a year ago, but got stuck on the Kubernetes piece.
Now that I've resolved that, I'm publishing it.</p>
<p>I've been a very happy user of <a href="https://saucelabs.com">SauceLabs</a>
for testing for many years. However, I don't
make a lot of use of it, and recently I've been trying to figure out how to cut down on
my dependence on external SaaS services (and, for that matter, some external paid-for
software). As part of this move, I decided to look at how I could run my
<a href="https://www.selenium.dev">selenium</a> tests without any third-party services.</p>
<p>I've beeen running the Selenium tests locally as part of the development test for years,
which has worked reasonably well (although be careful doing automated tests with
Safari, it's not well suited for testing on a machine that's also in active use).
However, using Chrome has been fine, and that works great when I need to run a quick
re-test before committing code to the repo and letting the CI run.</p>
<h2>SauceLabs as a workhorse</h2>
<p>However, for CI, I've had to put together some pretty complex cases. Generally, I don't
like exposing test systems to the internet when not necessary (at least prior to the
stage phase). As such, I the django test jig on one of my systems (frequently in
docker) and used SauceLab's <a href="https://docs.saucelabs.com/secure-connections/sauce-connect/">Sauce Connect Proxy</a>
to proxy back to my docker container and access the server running on localhost.
Despite the network delays involved in round-tripping to California for every test
command and going through a bespoke VPN, the process worked well and, save occasional
errors in communication, was a reliable testing environment.</p>
<p>Now, SauceLabs has a lot to recommend it, and it's got a really nice UI for teams (or
for that matter individuals), but the pricing model has changed over the years. When
I first signed up, I was paying $50/month for access, and that's remained the same as
I'm on a Legacy plan. However, I've been hesitant to move off of it because the Legacy
plans aren't available any longer and the remaining plans are a bit pricey for my use
case, starting at $149/month. To be fair, their pricing <em>model</em> changed and they're now
offering unlimited minutes for each parallel test. However, I don't need unlimited minutes.
In fact, I looked around and noticed that there are some other providers that handle
per-minute pricing, so originally I looked at that. However, the problem came back to
having to expose my system to the internet, unless I wanted to put up a proxy and
authentication, which just seemed unnecessarily annoying.</p>
<h2>Enter the Selenium Docker container</h2>
<p>I hadn't really looked at this in previous times because I wasn't running much docker
nor any kubernetes. Generally, my only docker was run locally on my Mac as a useful
way to fire up a linux environment when necessary.</p>
<p>This all changed last summer when I <a href="https://www.gaige.net/docker-on-smartos.html">moved to GitLab</a>,
leading me not only to run docker containers directly on my SmartOS infrastructure,
but also opening the door to Debian-based docker environments for testing.</p>
<p>With docker already running and GitLab already spinning up docker containers (and
containers in Kubernetes as well these days), I had a chance to look at the Selenium
Docker containers as a tool once again.</p>
<p>All told, there was a bit of experimentation to get it working, but I managed to get
it functioning in the docker environment with some small tweaks. In particular, I needed
to add:</p>
<div class="codehilite"><pre><span></span><code>--docker-shm-size 2000000000
</code></pre></div>
<p>to my runner (setting up for 2GB of shared memory), and setting a feature flag during
the run in order to ensure networking was happy (below). Once that was running, things
worked very well.</p>
<p>I also tried getting it running in my k8s runner environment, where I had less success,
until I finally set out to finish this article (nearly a year later). In the intervening
time, there were posts about the chrome shared memory exhaustion that was plaguing my
execution.</p>
<p>I need to keep tweaking this, but, from this
<a href="https://github.com/elgalu/docker-selenium/issues/20">issue</a> on docker-selenium, I got the
pointer I needed to
<a href="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614">stack overflow</a>
which recommended:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">spec</span><span class="p">:</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dshm</span>
<span class="w"> </span><span class="nt">emptyDir</span><span class="p">:</span>
<span class="w"> </span><span class="nt">medium</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Memory</span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gcr.io/project/image</span>
<span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/dev/shm</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dshm</span>
</code></pre></div>
<p>Which effectively translates into the following runner requirement in gitlab:</p>
<div class="codehilite"><pre><span></span><code> [[runners.kubernetes.volumes.empty_dir]]
name = "empty-dir"
mount_path = "/dev/shm"
medium = "Memory"
</code></pre></div>
<p>In the end, once I was past the shared memory and adjusted for networking
in the pod required localhost instead of the bridged network with local DNS, I was able
to stabilize the tests. And, it turns out they're a bit faster than the pure docker
environment tests.</p>
<p>In the end the tests were sufficiently unstable that I won't use them in a production
environment.</p>
<p>The CI/CD configuration is similar to the docker configuration with a few changes. This will
be detailed in the next section.</p>
<h2>Selenium-test configuration</h2>
<div class="codehilite"><pre><span></span><code><span class="nt">selenium-test</span><span class="p">:</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">docker</span><span class="p p-Indicator">,</span><span class="nv">selenium</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python:3.11</span>
<span class="w"> </span><span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FF_NETWORK_PER_BUILD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span>
<span class="w"> </span><span class="nt">GRID_URL</span><span class="p">:</span><span class="w"> </span><span class="s">"http://selenium__standalone-chrome:4444"</span>
<span class="w"> </span><span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">selenium/standalone-chrome:4</span>
<span class="c1"># alias: selenium</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="c1"># following is used for internal/sauceconnect use</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">apt-get update</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">apt-get install -y --no-install-recommends curl</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">curl -sSL https://install.python-poetry.org | python3 -</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">export PATH="~/.local/bin:$PATH"</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry install</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry run coverage erase</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mkdir -p output</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">curl $GRID_URL</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">poetry run coverage run --branch ./manage.py test selenium_tests</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mv .coverage .coverage-selenium-${CI_JOB_NAME}</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">always</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.coverage-selenium-${CI_JOB_NAME}</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">output</span>
<span class="w"> </span><span class="nt">reports</span><span class="p">:</span>
<span class="w"> </span><span class="nt">junit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">reports/junit.xml</span>
</code></pre></div>
<p>Using the <code>python:3.11</code> container to run the tests, along with a service container using the
<code>selenium/standalone-chrome</code> image as a sidecar in the docker environment to run the</p>
<p>The <code>FF_NETWORK_PER_BUILD</code> setting provides a
<a href="https://docs.gitlab.com/runner/configuration/feature-flags.html">private network</a>,
which prevents problems from interaction between multiple builds theoretically
running on the same machine (in my case, this would only happen in kubernetes).</p>
<p>The <code>GRID_URL</code> is created automatically by docker and results in the hostname for the
sidecar being <code>selenium__standalone-chrome</code> at port <code>4444</code>.</p>
<p>Beyond the variables, the rest is bootstrapping and running the tests that require
selenium. Since I'd been using it with Saucelabs, I don't have much to do to modify
the tests except make sure they're only using standard testing code (no need to send
telemetry information about builds to SauceLabs nowadays).</p>
<p>In this case, I load up poetry from the standard location, install it, set the path,
install my poetry app <code>poetry install</code> followed by <code>poetry run coverage erase</code> to make
sure there's no old image data.</p>
<p>Finally, I run the test through coverage, and rename the coverage results, so that they
don't overwrite the coverage results from my unit test step and can be combined.</p>
<p>Once the run is complete (regardless of the results), the coverage, output, and testing
reports are uploaded.</p>
<h3>Kubernetes modifications</h3>
<p>Since the kubernetes environment is a bit different, it uses a slightly modified
configuration, directly extending the docker configuration above:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">kube-test</span><span class="p">:</span>
<span class="w"> </span><span class="nt">extends</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">selenium-test</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">docker</span><span class="p p-Indicator">,</span><span class="nv">kubernetes</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">FF_NETWORK_PER_BUILD</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1</span>
<span class="w"> </span><span class="nt">KUBERNETES_SERVICE_CPU_REQUEST</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span>
<span class="w"> </span><span class="nt">KUBERNETES_SERVICE_MEMORY_REQUEST</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">4Gi</span>
<span class="w"> </span><span class="nt">GRID_URL</span><span class="p">:</span><span class="w"> </span><span class="s">"http://localhost:4444"</span>
</code></pre></div>
<p>The key configuration change here is the change in the <code>GRID_URL</code>, due to the difference
in network handling.</p>
<h2>Bonus round: combined coverage</h2>
<p>Since I've now got my unit tests and my UI tests running in the same pipeline,
I wanted to combine coverage, since the GitHub method of averaging doesn't really
represent full test coverage. If it were the same code, it'd give too much credit, but
if it were too little coverage, it'd not reresent it either. So what we really need
to do is pull both sets of coverage information and merge them. I do this using the
<code>coverage combine</code> command which is intended to combine multiple coverage files into the
same report. By doing this, the actual coverage of code is represented.</p>
<div class="codehilite"><pre><span></span><code><span class="nt">combine-coverage</span><span class="p">:</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">docker</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">report</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python:3.11</span>
<span class="w"> </span><span class="nt">needs</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">selenium-test</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">django-test</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python3 -m venv venv</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">source venv/bin/activate</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pip3 install coverage</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage combine .coverage-*</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mkdir -p reports</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage xml -o reports/coverage.xml</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">coverage html -d public</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">></span>
<span class="w"> </span><span class="no">grep ^\<coverage reports/coverage.xml</span>
<span class="w"> </span><span class="no">| sed -n -e 's/.*line-rate=\"\([0-9.]*\)\".*/\1/p'</span>
<span class="w"> </span><span class="no">| awk '{print "CodeCoverageOverall =" $1*100}'</span>
<span class="w"> </span><span class="no">|| true</span>
<span class="w"> </span><span class="nt">coverage</span><span class="p">:</span><span class="w"> </span><span class="s">'/^CodeCoverageOverall</span><span class="nv"> </span><span class="s">=(\d+\.\d+)$/'</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">always</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">public</span>
<span class="w"> </span><span class="nt">reports</span><span class="p">:</span>
<span class="w"> </span><span class="nt">coverage_report</span><span class="p">:</span>
<span class="w"> </span><span class="nt">coverage_format</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cobertura</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">reports/coverage.xml</span>
</code></pre></div>
<h2>Overall results</h2>
<p>Reliability of running in pure docker has been about as good as I've seen with SauceLabs
over the years, with much better control, and substantially lower costs. If your
environment allows, I'd encourage making use of it.</p>
Renovating GitLab Repos2023-04-10T06:35:00-04:002023-04-10T06:35:00-04:00Gaige B. Paulsentag:www.gaige.net,2023-04-10:/renovating-gitlab-repos.html<p>Over the past week, I've been working on getting my various dependencies
up to date in my GitLab instance repositories. The tool I'm using is
<a href="https://renovatebot.com">Mend Renovate</a>, an open-source solution
by the folks at <a href="https://mend.io">Mend</a> (formerly WhiteSource).</p>
<p>Let me state up front that I don't love the
<a href="https://github.com/renovatebot/renovate/blob/main/license">license</a> here,
it's …</p><p>Over the past week, I've been working on getting my various dependencies
up to date in my GitLab instance repositories. The tool I'm using is
<a href="https://renovatebot.com">Mend Renovate</a>, an open-source solution
by the folks at <a href="https://mend.io">Mend</a> (formerly WhiteSource).</p>
<p>Let me state up front that I don't love the
<a href="https://github.com/renovatebot/renovate/blob/main/license">license</a> here,
it's AGPL (formerly MIT for versions prior to version 12.0.0), but for my purposes,
it's OK since I'm not planning on modifying (other that potentially to submit bug
fixes or improvements) and I'm not providing the application as a service to others
(which is the key additional restriction).</p>
<p>Generally speaking, you're going to want a container environment for running
Renovate. Although you can run it using NPM, you're going to need an environment
in which the code and dependencies are available for inspection, so you need all
of the dependency-retrieving code available in your environment. For this, and
for isolation, I'm using containers.</p>
<p>Documentation for self-hosting Renovate in general is pretty extensive and is available for
<a href="https://docs.renovatebot.com/getting-started/running/#whitesource-renovate-on-premises">Self-hosted Mend Renovate</a>
online. Fair warning: there are a <em>lot</em> of knobs on this software.</p>
<h1>Bootstrapping renovate</h1>
<p>My first experiments involved using the well-maintained
<a href="https://gitlab.com/renovate-bot/renovate-runner">GitLab Runner</a>, that is freely available
and itself is updated by mend to the latest docker. The docs for the running in this configuration
are straightforward and provide a sufficient understanding of how to install in order to
get your first results quickly.</p>
<p>The recommendation is to use a dedicated private project to host the runner, and I concur
with that. I have a dedicated group for experiments like this and it fit well in that
location.</p>
<p>You'll need to define a number of CI/CD variables in order for it to work, but that's
straightforward and well documented.</p>
<p>Initially, I used the <code>RENOVATE_EXTRA_FLAGS</code> to specify individual projects instead
of using automated onboarding. As a rule, I found the explicit support to work well,
and wildcards were OK, but regex was very finicky, especially when using negatives
via the prefix <code>!</code> .</p>
<p>Make sure you put in a GitHub token as well as a GitLab token, since you will want to
have authenticated requests to GitHub in order to avoid the rate limits.</p>
<p>My final <code>.gitlab-ci.yml</code> used the <em>full</em> image in order to be able to handle a broader
array of dependencies and allowed for manual as well as scheduled operation:</p>
<div class="codehilite"><pre><span></span><code>include:
<span class="w"> </span>-<span class="w"> </span>remote:<span class="w"> </span>https://gitlab.com/renovate-bot/renovate-runner/-/raw/v12.13.0/templates/renovate.gitlab-ci.yml
image:<span class="w"> </span><span class="cp">${</span><span class="n">CI_RENOVATE_IMAGE_FULL</span><span class="cp">}</span>
renovate:
<span class="w"> </span>rules:
<span class="w"> </span>-<span class="w"> </span>if:<span class="w"> </span><span class="nv">$CI_PIPELINE_SOURCE</span><span class="w"> </span>==<span class="w"> </span>"schedule"
<span class="w"> </span>-<span class="w"> </span>if:<span class="w"> </span><span class="nv">$CI_PIPELINE_SOURCE</span><span class="w"> </span>==<span class="w"> </span>"web"
<span class="w"> </span>-<span class="w"> </span>if:<span class="w"> </span><span class="nv">$CI_PIPELINE_SOURCE</span><span class="w"> </span>==<span class="w"> </span>"push"
</code></pre></div>
<p>Note: you may want to renovate to keep the script version up-to-date.</p>
<p>Once you've got this running, individual repositories carry their configurations in
<code>renovate.json</code> files (which may be stored in a variety of locations, I generally put
mine in <code>.gitlab/renovate.json</code>). When not present, the repository is ignored.</p>
<h1>Permissions and users</h1>
<p>While it is possible to run Renovate as yourself, especially when you're only running it
on your repositories, there is less confusion if you have a separate bot user dedicated to
providing these updates.</p>
<p>By using a separate user, you can fully scope access, which is especially important
if you have admin access to your repositories or have access to a wider array of repos than
you want to give Renovate access to.</p>
<p>Further, if you decide to go with Mend Renovate On-Premises, you'll find that the webhooks
are basically ineffective if it can't distinguish between your actions and its own by the
user making the change.</p>
<p>In the end, I thought it was definitely worth it, because I was also able to enable
autodiscovery since each repo (or group) was intentionally onboarded to Renovate.</p>
<h1>Mend Renovate On-Premises</h1>
<p>There's also a version of Renovate that's designed to run with its own scheduler and
responds to webhooks from GitLab. This version of
<a href="https://github.com/mend/renovate-on-prem">Mend Renovate On-Premises</a>
is what I've moved to over the weekend. This one definitely wants a stable container
environment to run in, and one in which you'll need to be able to communicate between
your source control system (in my case GitLab) and the renovate server (otherwise, you
won't get the benefits of the webhooks, and might as well stick with the easier-to-manage
cron-only version above).</p>
<p>This is licensed software from Mend, and requires a key in order to run. The keys are
available <a href="https://www.mend.io/free-developer-tools/renovate/on-premises/">for Free</a>
once you request them. It took a little time, and you should expect to be prospected for
sales, but that's certainly fair.</p>
<p>Once the key is received, you'll need to prepare to configure your environment. In my
case, this is a small kubernetes environment. There's a Helm chart that is mostly
well documented in the aforementioned repository, that goes through the basics,
installation with Helm and configuration for both GitHub and GitLab—in both cases
either self-hosted or SaaS.</p>
<p>Setting up the environment via Helm was straightforward and I created an application
in my gitlab-managed environment, using <code>.gotmpl</code> in order to fill in my secret information
using the GitLab CI/CD variables and runtime information.</p>
<p>My <code>helmfile.yaml</code> looks like this:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">repositories</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">renovate-on-prem</span>
<span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://mend.github.io/renovate-on-prem</span>
<span class="nt">releases</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">renovate</span>
<span class="w"> </span><span class="nt">namespace</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">YOUR-NAMESPACE-HERE</span>
<span class="w"> </span><span class="nt">chart</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">renovate-on-prem/whitesource-renovate</span>
<span class="w"> </span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3.1.3</span>
<span class="w"> </span><span class="nt">installed</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">values</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">values.yaml.gotmpl</span>
</code></pre></div>
<p>which basically adds the chart repo and then runs a specific version based on the
chart available and using the included values file.</p>
<p>My <code>values.yaml.gotmpl</code> is a bit more complex:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">credentials</span><span class="p">:</span>
<span class="w"> </span><span class="nt">gitlab_access_token</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_GITLAB_TOKEN"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">github_access_token</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_GITHUB_TOKEN"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="nt">renovate</span><span class="p">:</span>
<span class="w"> </span><span class="nt">acceptWhiteSourceTos</span><span class="p">:</span><span class="w"> </span><span class="s">'y'</span>
<span class="w"> </span><span class="nt">licenseKey</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_LICENSE_KEY"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">renovatePlatform</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gitlab</span>
<span class="w"> </span><span class="nt">renovateEndpoint</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "GITLAB_API_URL"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">renovateToken</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_GITLAB_TOKEN"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">githubComToken</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_GITHUB_TOKEN"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">webhookSecret</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">{{</span><span class="w"> </span><span class="nv">requiredEnv "RENOVATE_WEBHOOK_TOKEN"</span><span class="w"> </span><span class="p p-Indicator">}}</span>
<span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">module.exports = {</span>
<span class="w"> </span><span class="no">"autodiscoverFilter": ["!/{data,experiment,imported}/.*/"],</span>
<span class="w"> </span><span class="no">"packageRules": [</span>
<span class="w"> </span><span class="no">{</span>
<span class="w"> </span><span class="no">"matchUpdateTypes": ["major", "minor", "patch", "digest", "bump"],</span>
<span class="w"> </span><span class="no">"addLabels": ["dependencies"]</span>
<span class="w"> </span><span class="no">},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["ruby"], "addLabels": ["ruby"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["java"], "addLabels": ["java"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["python"], "addLabels": ["python"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["php"], "addLabels": ["php"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["js"], "addLabels": ["js"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["docker"], "addLabels": ["docker"]},</span>
<span class="w"> </span><span class="no">{"matchLanguages": ["git-submodules"], "addLabels": ["submodule"]}</span>
<span class="w"> </span><span class="no">],</span>
<span class="w"> </span><span class="no">"git-submodules": {"enabled": true},</span>
<span class="w"> </span><span class="no">"hostRules": [</span>
<span class="w"> </span><span class="no">{ "matchHost": "{{ requiredEnv "CI_SERVER_HOST" }}", "token":"{{ requiredEnv "RENOVATE_GITLAB_TOKEN" }}" },</span>
<span class="w"> </span><span class="no">{ "matchHost": "{{ requiredEnv "CI_SERVER_HOST" }}", "hostType": "docker", "username": "token", "password":"{{ requiredEnv, "RENOVATE_GITLAB_TOKEN" }}" },</span>
<span class="w"> </span><span class="no">{ "matchHost": "{{ requiredEnv "CI_SERVER_HOST" }}", "hostType": "helm", "username": "token", "password":"{{ requiredEnv "RENOVATE_GITLAB_TOKEN" }}" }</span>
<span class="w"> </span><span class="no">],</span>
<span class="w"> </span><span class="no">"dependencyDashboard":"true",</span>
<span class="w"> </span><span class="no">"dependencyDashboardLabels": ["dashboard"]</span>
<span class="w"> </span><span class="no">}</span>
<span class="nt">podSecurityContext</span><span class="p">:</span>
<span class="w"> </span><span class="nt">fsGroup</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1000</span>
<span class="nt">cachePersistence</span><span class="p">:</span>
<span class="w"> </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">storageClass</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">longhorn</span>
<span class="nt">ingress</span><span class="p">:</span>
<span class="w"> </span><span class="nt">enabled</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">ingressClassName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">nginx</span>
<span class="w"> </span><span class="nt">annotations</span><span class="p">:</span>
<span class="w"> </span><span class="nt">cert-manager.io/cluster-issuer</span><span class="p">:</span><span class="w"> </span><span class="s">"YOUR_CERT_ISSUER"</span>
<span class="w"> </span><span class="nt">nginx.ingress.kubernetes.io/whitelist-source-range</span><span class="p">:</span><span class="w"> </span><span class="s">"YOUR_CI_SERVER"</span>
<span class="w"> </span><span class="nt">hosts</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">YOUR_RENOVATE_HOSTNAME</span>
<span class="w"> </span><span class="nt">tls</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">secretName</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">renovate-fe-cert</span>
<span class="w"> </span><span class="nt">hosts</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">YOUR_RENOVATE_HOSTNAME</span>
</code></pre></div>
<p>Before using this, take a good look at it. There are some common items here, and some
lessons learned.</p>
<ul>
<li>If you don't set the <code>fsGroup</code> and you use the cache, you may find that the cache
is not writable. The application runs as UID/GID 1000, so this setting just makes
sure the application can write to it.</li>
<li>I'm using Longhorn for my local storage, so <code>storageClass</code> is set to that. If you're
using something different, take appropriate changes</li>
<li>For getting the webhooks, I am using an nginx ingress controller, metallb, and
cert-manager with support for ACME. I constrained the access to the ingress server
using the <code>nginx.ingress.kubernetes.io/whitelist-source-range</code> annotation. TLS is
configured and my gitlab server is checking for appropriate TLS certificates.</li>
<li>The weird <code>hostRules</code> lines are because I need to authenticate back to the gitlab server
for a couple of repository types. I've pulled this discussion into
<a href="https://www.gaige.net/renovating-gitlab-registries.html">Renvating GitLab Registries</a> after
finding some further nuance.</li>
<li>The documentation on Renovate has a lot of settings. There are multiple ways to add
default changes. In this case, I chose to force settings by applying them globally. You
can also do these locally using <code>renovate.json</code> files (which will be sought out as the
indicator that you want Renovate to run in those repos), and by specifying global
defaults. Read the documentation and look for best practices.</li>
<li>The defaults mostly worked, but I like to have my PRs labeled, so I added the various
<code>packageRules</code> in order to set labels.</li>
<li>I find the <a href="https://docs.renovatebot.com/key-concepts/dashboard/">Dependency Dashboard</a>
to be useful, so I configured it (and also set
a label for it, so I could do a an easy global search for a sort-of dashboard of
dashboards).</li>
</ul>
<p>Although the helm chart seems up-to-date for the most part, I did eventually start
playing with the image tags in order to get a more up-to-date version of the underlying
image. Do this at your own peril. There's nothing theoretically wrong with it, as they
are released at least in some fashion, but you may want to stick with the standards.</p>
<p>If you do decide to go with the modified image:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">image</span><span class="p">:</span>
<span class="w"> </span><span class="nt">repository</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">whitesource/renovate</span>
<span class="w"> </span><span class="nt">tag</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">VERSION</span>
<span class="w"> </span><span class="nt">pullPolicy</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">IfNotPresent</span>
</code></pre></div>
<p>is what I added to retrieve the latest renovate image, and you'll need to visit
Docker Hub in order to get the latest tag.</p>
<h1>Results</h1>
<p>All told, I'm happy with the implementation. It took a little while to bootstrap, but
it is doing a good job at finding my updates and providing a pretty uncluttered and
automated mechanism for keeping up to date.</p>
Bacula pruning old storage2022-10-05T07:22:00-04:002022-10-05T07:22:00-04:00Gaige B. Paulsentag:www.gaige.net,2022-10-05:/bacula-pruning-old-storage.html<p>I note with some amusement the fact that I wrote on this exact day last year about
this same subject (in much more detail).</p>
<p>The reason for the new message on this subject is that I'm still cleaning up some of the
decisions I made when first using Bacula.</p>
<h2>The …</h2><p>I note with some amusement the fact that I wrote on this exact day last year about
this same subject (in much more detail).</p>
<p>The reason for the new message on this subject is that I'm still cleaning up some of the
decisions I made when first using Bacula.</p>
<h2>The Problem</h2>
<p>Before I realized that I really needed to have three pools to make things work correctly,
I was storing the full, differential, and incremental backups in the same pool. This
turned out to be a bad decision, and I rectified it prior to my last note on the topic—
however, not until I'd accumulated a lot of "volumes" of data that were still in my
main pool (the one I continue to use for my full backups).</p>
<p>If you look at volumes the way that Bacula does, they're basically tapes. As such,
they're considered not to take up any more space when they're full as when they're empty.
This makes sense for tapes, since the tendency in a tape-based system is to rotate media
physically through offsite and onsite storage and then to tape libraries or robots for
reuse.
For on-disk volumes, though, this poses a different problem—empty on-disk volumes and
full ones do not take up the same amount of space.</p>
<p>In my case, I was running low on disk space on my storage volume and when I looked I
noticed many volumes that had not been written to in quite some time and were marked
as "expired". These volumes would eventually be reused, but since that pool had run
for quite some time with full, differential, and incremental backups in it, I had a
large number of "tapes" that contained expired data to write over. As the full backups
take a relatively constant number of volumes, they would take a few more years to
overwrite the volumes used for differential and incremental backups.</p>
<p>In short, I had a lot of <em>used</em> volumes taking up space (both in the catalog and in
storage) that should have been purged.</p>
<h2>The Solution</h2>
<p>In order to stop bacula from continuing to allow the "old" storage to lie around,
I needed to delete the volumes. This makes sense if you think like tape—once you've bought
the tape, you might as well leave the data on it (ignoring the security concerns) until
you need to reuse it. But, this isn't tape, and there's a significant benefit to keeping
the number of volumes right-sized for our environment.</p>
<p>In my case, I wanted to remove all volumes from my <code>Cloud-CT</code> media pool that
were more than 2 years old. In this case, the bacula <code>sql</code> command came in handy as
I was able to directly query the database:</p>
<div class="codehilite"><pre><span></span><code><span class="k">select</span><span class="w"> </span><span class="s1">'prune yes volume='</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="n">volumename</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">media</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">lastwritten</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="s1">'2020-10-03'</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="n">mediatype</span><span class="o">=</span><span class="s1">'Cloud-CT'</span><span class="w"> </span><span class="k">order</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">volumename</span><span class="p">;</span>
</code></pre></div>
<p>The above query resulted in a number of lines that could be pasted directly into the
bconsole in order to purge the volumes.</p>
<p>Based on some examples, and out of an abundance of caution, I decided to purge the volumes
before deleting them. This was likely an unnecessary step, but it ensured that my catalog
database was cleaned as well.</p>
<p>Once done, I was able to rerun the sql command, replacing <code>prune</code> with <code>delete</code> to delete
the unnecessary volumes.</p>
<p>This cleared up all the near-side volumes, removing the storage that they consumed as
well as their markers so that they would not be reused in the future.</p>
<p>For the far-side (cloud) copies, I opted to directly purge those using a find command:</p>
<div class="codehilite"><pre><span></span><code>find<span class="w"> </span>bacula-west/<span class="w"> </span>-mtime<span class="w"> </span>+720<span class="w"> </span>-exec<span class="w"> </span>rm<span class="w"> </span>-r<span class="w"> </span><span class="se">\{\}</span><span class="w"> </span><span class="se">\;</span>
</code></pre></div>
<p>Where bacula-west is the name of the storage location.</p>
<h2>Summary</h2>
<p>All told, if I'd known originally what a mess I was creating by using a single pool, I
would have resolved that earlier, but this is how we learn.</p>
Test without building and SPM2022-05-17T07:45:00-04:002022-05-17T07:45:00-04:00Gaige B. Paulsentag:www.gaige.net,2022-05-17:/test-without-building-and-spm.html<p>Another day, another set of testing issues. As mentioned in my previous post,
<a href="https://www.gaige.net/slathering-xcode-variants.html">Slathering Xcode Variants</a>,
I've been making some use of Xcode's capability to build a test package and
separately run that test package on a different machine, possibly with a different
version of macOS or even a different …</p><p>Another day, another set of testing issues. As mentioned in my previous post,
<a href="https://www.gaige.net/slathering-xcode-variants.html">Slathering Xcode Variants</a>,
I've been making some use of Xcode's capability to build a test package and
separately run that test package on a different machine, possibly with a different
version of macOS or even a different CPU architecture.</p>
<p>In order to meet my testing needs, I've generally been building with the latest Xcode
on the latest version of macOS and then running tests on both that version and the
previous version of macOS. I've been running in this mode for months, without any
difficulty, until this week.</p>
<p>Earlier this week, I added a new SPM dependency to one of the Xcode projects in my
workspace. Since this project is large, the organization is a single workspace with
multiple projects in it (today that number is 16) and a handful of SPM dependencies
(3 across the whole set today).</p>
<p>After adding the dependency and running full tests on my Mac Pro (x86_64, macOS Monterey),
things were looking good, so I pushed the new files to my CI server. A few minutes later,
my Big Sur tests both failed. Looking at the logs, I found:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">Package</span>.<span class="nv">resolved</span><span class="w"> </span><span class="nv">file</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">corrupted</span><span class="w"> </span><span class="nv">or</span><span class="w"> </span><span class="nv">malformed</span><span class="c1">; fix or delete the file to continue: unsupported schema version 2</span>
</code></pre></div>
<p>This only happend on the Big Sur machines, and they are running Xcode 13.2.x (last
version before 13.3 came along and stopped running on Big Sur).</p>
<p>Not surprisingly, this is because this file was created on my Monterey Mac.</p>
<h2>Trying without Fastlane</h2>
<p>I'll caution that these are all still running within Fastlane, so it's possible that I'm
shooting myself in the foot by not pushing all the parameters to the command line
manually. I may, at some point, give that a try and see if that solves the problem.</p>
<p>Doing some manual testing confirmed that when not using Fastlane, I could easily skip the
problematic code when running in CI by targeting the <code>xctestrun</code> file directly instead
of using the Scheme and Workspace (thus avoiding the workspace evaluation).</p>
<p>For example:</p>
<div class="codehilite"><pre><span></span><code>xcodebuild<span class="w"> </span>test-without-building<span class="w"> </span>⏎
<span class="w"> </span>-xctestrun<span class="w"> </span>test_build/Build/Products/Cartographica_Cartographica-Exhaustive_macosx12.3-arm64-x86_64.xctestrun<span class="w"> </span>⏎
<span class="w"> </span>-destination<span class="w"> </span><span class="s1">'platform=macOS'</span><span class="w"> </span>⏎
<span class="w"> </span>-resultBundlePath<span class="w"> </span><span class="s1">'output/Cartographica.xcresult'</span>
</code></pre></div>
<p>(once again using ⏎ to denote, counter-intuitively, that the line breaks here are only
for readability and should be left out).</p>
<p>This command explicitly uses the <code>-xctestrun</code> option, pointing it at the specific <code>xctestrun</code>
file instead of using</p>
<h2>Back to Fastlane</h2>
<p>Eventually, I decided I still wanted to use Fastlane (although I feel like I'm
using few enough features that there may be a future blog post about kicking it
to the curb as well).</p>
<p>I'm having to be more careful about how I build and run my tests (notably: needing to
make sure I don't bump the test libraries too far). However, things are working and
the process survived at least the initial jump to Ventura.</p>
<p>My build and test matrix now looks like this:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">build-mac</span><span class="p">:</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build</span>
<span class="w"> </span><span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">GIT_CLONE_PATH</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${CI_BUILDS_DIR}/${CI_PROJECT_PATH}</span>
<span class="w"> </span><span class="nt">before_script</span><span class="p">:</span><span class="w"> </span><span class="nv">*mac_build_prep</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">xcode14</span><span class="p p-Indicator">,</span><span class="nv">codesigning</span><span class="p p-Indicator">,</span><span class="nv">build</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">needs</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">check-servers</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bundle exec fastlane --verbose cibuild</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test_build/Build/Products</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test_build/Build/SourcePackages</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test_build/Logs</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test_build/info.plist</span>
<span class="w"> </span><span class="nt">expire_in</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1 day</span>
<span class="nt">.test_template</span><span class="p">:</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="w"> </span><span class="nt">needs</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">job</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build-mac</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">GIT_CLONE_PATH</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${CI_BUILDS_DIR}/${CI_PROJECT_PATH}</span>
<span class="w"> </span><span class="nt">CTRunningUnderTest</span><span class="p">:</span><span class="w"> </span><span class="s">'YES'</span>
<span class="w"> </span><span class="nt">coverage</span><span class="p">:</span><span class="w"> </span><span class="s">'/CodeCoverageOverall</span><span class="nv"> </span><span class="s">=\d+\.\d+/'</span>
<span class="w"> </span><span class="nt">interruptible</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">before_script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">export PATH=~/.rbenv/shims:${PATH}</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">export FL_SLATHER_ARCH=`uname -m`</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo $PATH</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bundle install</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">rm -rf ${CI_PROJECT_DIR}/DerivedData/Build/ProfileData</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">rm -rf ${CI_PROJECT_DIR}/output/*</span>
<span class="w"> </span><span class="nt">artifacts</span><span class="p">:</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">always</span>
<span class="w"> </span><span class="nt">reports</span><span class="p">:</span>
<span class="w"> </span><span class="nt">junit</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">output/report.junit</span>
<span class="w"> </span><span class="nt">coverage_report</span><span class="p">:</span>
<span class="w"> </span><span class="nt">coverage_format</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cobertura</span>
<span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">output/cobertura.xml</span>
<span class="w"> </span><span class="nt">paths</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">output/scan/*.log</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">output/*.xcresult</span>
<span class="nt">.coverage_script</span><span class="p">:</span><span class="w"> </span><span class="nl">&coverage_script</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="p p-Indicator">></span>
<span class="w"> </span><span class="no">grep ^\<coverage output/cobertura.xml</span>
<span class="w"> </span><span class="no">| sed -n -e 's/.*line-rate=\"\([0-9.]*\)\".*/\1/p'</span>
<span class="w"> </span><span class="no">| awk '{print "CodeCoverageOverall =" $1*100}'</span>
<span class="w"> </span><span class="no">|| true</span>
<span class="nt">test</span><span class="p">:</span>
<span class="w"> </span><span class="nt">extends</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.test_template</span>
<span class="w"> </span><span class="nt">parallel</span><span class="p">:</span>
<span class="w"> </span><span class="nt">matrix</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">PROCESSOR</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">arm64</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">OS</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bigsur</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">PROCESSOR</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">arm64</span><span class="p p-Indicator">,</span><span class="nv">x86_64</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">OS</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">monterey</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">PROCESSOR</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">arm64</span><span class="p p-Indicator">,</span><span class="nv">x86_64</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">OS</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ventura</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">codesigning</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${OS}</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">${PROCESSOR}</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bundle exec fastlane --verbose citest</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nv">*coverage_script</span>
</code></pre></div>
<p><em>Ed. Note</em>: Updated for ventura</p>
<p>And my fastfile for running the tests (without building) uses a number of different
commands:</p>
<ul>
<li><code>test_from_build</code> (to run the tests based on the testplan)</li>
<li><code>build_for_tests</code> (to create the binaries for running the tests)</li>
</ul>
<p>Neither of these are called directly, but are called from the <code>cibuild</code> and <code>citest</code> above</p>
<p>Note that <code>build_for_tests</code> takes an argument of an array of testplans to build.</p>
<div class="codehilite"><pre><span></span><code><span class="n">default_platform</span><span class="p">(</span><span class="ss">:mac</span><span class="p">)</span>
<span class="n">my_xcargs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span>
<span class="n">platform</span><span class="w"> </span><span class="ss">:mac</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="n">desc</span><span class="w"> </span><span class="s1">'Build for testing'</span>
<span class="w"> </span><span class="n">lane</span><span class="w"> </span><span class="ss">:build_for_tests</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">options</span><span class="o">|</span>
<span class="w"> </span><span class="n">final_xcargs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span><span class="n">my_xcargs</span><span class="p">,</span><span class="w"> </span><span class="s1">'ONLY_ACTIVE_ARCH=NO'</span><span class="o">].</span><span class="n">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:testplan</span><span class="o">]</span>
<span class="w"> </span><span class="n">final_xcargs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="o">[</span><span class="n">my_xcargs</span><span class="p">,</span><span class="w"> </span><span class="s1">'ONLY_ACTIVE_ARCH=NO'</span><span class="o">]+</span><span class="n">args_with_prefix</span><span class="p">(</span><span class="n">options</span><span class="o">[</span><span class="ss">:testplan</span><span class="o">]</span><span class="p">,</span><span class="s1">'-testPlan '</span><span class="p">))</span><span class="o">.</span><span class="n">reject</span><span class="p">(</span><span class="o">&</span><span class="ss">:empty?</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
<span class="w"> </span><span class="k">end</span>
<span class="w"> </span><span class="n">run_tests</span><span class="p">(</span><span class="ss">scheme</span><span class="p">:</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:scheme</span><span class="o">]</span><span class="p">,</span>
<span class="w"> </span><span class="ss">configuration</span><span class="p">:</span><span class="w"> </span><span class="s1">'Debug'</span><span class="p">,</span>
<span class="w"> </span><span class="ss">code_coverage</span><span class="p">:</span><span class="w"> </span><span class="kp">true</span><span class="p">,</span>
<span class="w"> </span><span class="ss">address_sanitizer</span><span class="p">:</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span>
<span class="w"> </span><span class="ss">output_types</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span>
<span class="w"> </span><span class="ss">disable_slide_to_type</span><span class="p">:</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span><span class="w"> </span><span class="c1"># note: this gets around a macos bug caused by assuming ios in fastlane</span>
<span class="w"> </span><span class="ss">clean</span><span class="p">:</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:clean</span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span>
<span class="w"> </span><span class="ss">xcargs</span><span class="p">:</span><span class="w"> </span><span class="n">final_xcargs</span><span class="p">,</span>
<span class="w"> </span><span class="ss">derived_data_path</span><span class="p">:</span><span class="w"> </span><span class="s2">"test_build"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">build_for_testing</span><span class="p">:</span><span class="w"> </span><span class="kp">true</span><span class="p">)</span>
<span class="w"> </span><span class="k">end</span>
<span class="w"> </span><span class="n">desc</span><span class="w"> </span><span class="s1">'Runs built tests'</span>
<span class="w"> </span><span class="n">lane</span><span class="w"> </span><span class="ss">:test_from_build</span><span class="w"> </span><span class="k">do</span><span class="w"> </span><span class="o">|</span><span class="n">options</span><span class="o">|</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:testplan</span><span class="o">]</span>
<span class="w"> </span><span class="n">final_xcargs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="o">[</span><span class="n">my_xcargs</span><span class="o">]+</span><span class="n">args_with_prefix</span><span class="p">(</span><span class="n">options</span><span class="o">[</span><span class="ss">:testplan</span><span class="o">]</span><span class="p">,</span><span class="s1">'-testPlan '</span><span class="p">))</span><span class="o">.</span><span class="n">reject</span><span class="p">(</span><span class="o">&</span><span class="ss">:empty?</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="n">final_xcargs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">my_xcargs</span>
<span class="w"> </span><span class="k">end</span>
<span class="w"> </span><span class="n">run_tests</span><span class="p">(</span><span class="ss">scheme</span><span class="p">:</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:scheme</span><span class="o">]</span><span class="p">,</span>
<span class="w"> </span><span class="ss">configuration</span><span class="p">:</span><span class="w"> </span><span class="s1">'Debug'</span><span class="p">,</span>
<span class="w"> </span><span class="ss">code_coverage</span><span class="p">:</span><span class="w"> </span><span class="kp">true</span><span class="p">,</span>
<span class="w"> </span><span class="ss">address_sanitizer</span><span class="p">:</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span>
<span class="w"> </span><span class="ss">output_types</span><span class="p">:</span><span class="w"> </span><span class="s2">"junit"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">disable_slide_to_type</span><span class="p">:</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span><span class="w"> </span><span class="c1"># note: this gets around a macos bug caused by assuming ios in fastlane</span>
<span class="w"> </span><span class="ss">clean</span><span class="p">:</span><span class="w"> </span><span class="n">options</span><span class="o">[</span><span class="ss">:clean</span><span class="o">]</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="kp">false</span><span class="p">,</span>
<span class="w"> </span><span class="ss">xcargs</span><span class="p">:</span><span class="w"> </span><span class="n">final_xcargs</span><span class="p">,</span>
<span class="w"> </span><span class="ss">derived_data_path</span><span class="p">:</span><span class="w"> </span><span class="s2">"test_build"</span><span class="p">,</span>
<span class="w"> </span><span class="ss">output_directory</span><span class="p">:</span><span class="w"> </span><span class="s1">'output'</span><span class="p">,</span>
<span class="w"> </span><span class="ss">test_without_building</span><span class="p">:</span><span class="w"> </span><span class="kp">true</span><span class="p">)</span>
<span class="w"> </span><span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
Slathering Xcode variants2022-05-10T06:23:00-04:002022-05-10T06:23:00-04:00Gaige B. Paulsentag:www.gaige.net,2022-05-10:/slathering-xcode-variants.html<p>I've been doing quite a bit of experimentation with recent features in Xcode lately,
especially as regards trying to efficiently run my GitLab-powered Mac Mini build farm.</p>
<p>Recently, as I've been doing some work on CartoMobile, I've been updating the testing
code there and stole some ideas from the Cartographica …</p><p>I've been doing quite a bit of experimentation with recent features in Xcode lately,
especially as regards trying to efficiently run my GitLab-powered Mac Mini build farm.</p>
<p>Recently, as I've been doing some work on CartoMobile, I've been updating the testing
code there and stole some ideas from the Cartographica test suites, which I intentionally
build for both Apple Silicon and x86_64 and then run on 2 OS variants with each processor
family. In this case, I collect coverage information from all 4 and then merge them because
I have variant code that runs on different CPUs and versions of the OS (more the former
than the later, because some libraries are specific to one architecture or the other).</p>
<p>In Cartographica, the CI code follows these steps:</p>
<ol>
<li>
<p>Build the code for testing</p>
</li>
<li>
<p>Run a matrix job across the CPUs and Operating Systems that I need to test,
collecting junit and coverage information</p>
<ul>
<li>Current macOS and x86_64</li>
<li>Current macOS and arm64</li>
<li>Previous macOS and x86_64</li>
<li>Previous macOS and arm64</li>
</ul>
</li>
</ol>
<p>For CartoMobile, I did something similar, but ran into a problem with <code>slather</code> in doing
so and also realized that I was likely wasting time and effort.</p>
<p>First, the problem that I ran into was specifically with running slather without pointing
at the correct directory. In this case, I wasn't pointing slather at the migrated
directory when checking coverage, thus failing to find the coverage files when running.</p>
<p>However, more importantly, this led me to the realization that the way that I was going
about this was wrong for CartoMobile. Unlike Cartographica, where I was doing matrixed
coverage because I have code that only operates on specific CPU architectures or
versions of macOS, CartoMobile has a single set of code that runs on all SDKs,
and since I can't run coverage tests on the <code>iphoneos</code> SDK, that meant that all that
was interesting was running the coverage tests for the simulator on both iPadOS and iOS.</p>
<p>In addition, for CartoMobile, I also run TSAN and ASAN tests (with UBSAN set). Although
the builds for those can take a while, the runtimes are short, and the builds are
completely independent. Further, the coverage for these are not important, since
coverage tests
can (and should) be run without the sanitizers. Thus, unlike Cartographica,
for CartoMobile, I decided to matrix the build and test functions together.
The result is:</p>
<ol>
<li>
<p>Build for simulator and test with Coverage, collecting junit and coverage</p>
</li>
<li>
<p>Build and run simulator using an Xcode test plan (.xctestplan) that runs with:</p>
<ul>
<li>TSAN + UBSAN</li>
<li>ASAN + UBSAN</li>
</ul>
</li>
<li>
<p>Build and run my iOS snapshots on a minimum iPad and iPhone simulator (UI Test)</p>
</li>
<li>
<p>Build and run my iOS snapshots on all sizes and languages (AppStore snapshots)</p>
</li>
</ol>
Renaming Elasticsearch indexes2022-04-04T06:23:00-04:002022-04-04T06:23:00-04:00Gaige B. Paulsentag:www.gaige.net,2022-04-04:/renaming-elasticsearch-indexes.html<p>I've been an <a href="https://www.elastic.co/what-is/elk-stack">ELK Stack</a>
(Elasticsearch, Logstash, Kibana, and Beats) user for quite some time, using
exclusively the open source version fo the stack.</p>
<p>Generally it's works well and, with some exceptions, supports our mostly-Solaris
based environment (using LX zones to host most of the beefier components, and
using custom-built …</p><p>I've been an <a href="https://www.elastic.co/what-is/elk-stack">ELK Stack</a>
(Elasticsearch, Logstash, Kibana, and Beats) user for quite some time, using
exclusively the open source version fo the stack.</p>
<p>Generally it's works well and, with some exceptions, supports our mostly-Solaris
based environment (using LX zones to host most of the beefier components, and
using custom-built beats and senders for the lightweight senders).</p>
<h2>Index Lifecycle Management (ILM)</h2>
<p>A couple of years ago, I started using ILM, which automatically rotates indexes
through various
<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html">stages of life</a>
based on use of the index:</p>
<ul>
<li>hot: index is being actively updated and queried</li>
<li>warm: index is not being updated, but is still being queried</li>
<li>cold: no longer being updated, but queried infrequently</li>
<li>frozen: no longer being updated, queried rarely</li>
<li>delete: no longer needed and may be deleted</li>
</ul>
<p>Generally, you set up a file pattern which is followed automatically for the indexes
creating a user-and-machine-friendly <code>YYYY-MM-DD-NNN</code> suffix, such as
<code>mail-7.4.2-2020-04-22-0027</code> so that the name denotes its creation date and has
a numeric discriminator in case you need to rotate due to volume instead of dates.
ILM supports rotation based on a number of factors, and it's common to specify a
monthly rotation plus a file size rotation, in order to keep indexes a manageable size.</p>
<h2>Fixing a bad index template</h2>
<p>During one of my index creation steps, I used the wrong name template and ended up
with all of my subsequent indexes (for over a year) being named with the same date
and an incrementing discriminator value. This was practically fine, but annoying
because it made it difficult to determine if the indexes were being deleted and rotated
through stages correctly.</p>
<p>The fix for this was to make sure that the index being written by my logstash component
was <code><mail-8.0.0-{now/d}-000032></code> (as an example), not <code>mail-8.0.0-{now/d}-000032</code>; the
difference being when the value was evaluated. In the former, it's kept with the index;
in the latter, it's evaluated when the index is created and the result is a
<code>provided_name</code> of <code>mail-8.0.0-</code><em><code>date-created</code></em><code>-000032</code>, which means the date won't
change.</p>
<p>I tested this change by modifying the <code>provided_name</code> of the running index, and then
executing the manual rollover by sending:</p>
<div class="codehilite"><pre><span></span><code>POST <rollover-target>/_rollover/<target-index>
</code></pre></div>
<p>and specifying the new index template in the <code><target-index></code> and then subsequently
setting <code>index.lifecycle.indexing_complete</code> to <code>true</code> on the index so that the lack
of an <a href="https://discuss.elastic.co/t/index-management-error-index-is-not-the-write-index-for-alias/180894/2">automated rollover didn't cause error messages</a>.</p>
<h2>Changing historic names</h2>
<p>The remaining problem was the names of the old indexes. Although w does not
have a <em>rename</em> command, it does have two other useful commands, <code>reindex</code> and <code>clone</code>.</p>
<p><code>reindex</code> will reindex all of the documents in the original index into a new
index, which allows you to change the format of the index and settings prior to reindexing
the data (in fact, you must create and provision the new index first, or you're likely
going to either delete the reindexed copy or re-reindex it).</p>
<p><code>clone</code> makes a complete clone of the index by using hard links (if possible on the
underlying OS). The makes it particularly fast (at least for the primary shards) and
allows you to create the new index with all of the attributes of the old index.</p>
<p>So, for the equivalent of renaming the indexes, you <code>clone</code> the old index to the new
name, and then <code>delete</code> the original. In this case, you're only rebuilding the replicas
and by deleting the old index, the hard links become referenced only by the new index.</p>
<h2>Setting the ILM time</h2>
<p>The final problem was that all of these indexes now were believed by ILM to be brand
new. They were going to get rotated into another phase (or deleted) based on the
date that I cloned them, not based on the last write date.</p>
<p>As an aside: the way that ILM looks at indexes is by considering the creation date to
be the key date for the index (the <code>lifecycle_date_millis</code>) to be the creation date of
the index, until it is closed for the first rollover, at which point the
<code>lifecycle_date_millis</code> is set to that first rollover date. This way, ILM actions are based
on the creation date of the index (rollover 30d after creation, for example) and subsequent
actions are made based on the date that index was closed.</p>
<p>By cloning the index, I'd reset the creation date, and thus the <code>lifecycle_date_millis</code>.
Not surprisingly, this was a pretty easy fix: determine the rollover date and then
reset the value.</p>
<p>In my case, I double-checked the expected dates by executing a timestamp query:</p>
<div class="codehilite"><pre><span></span><code>GET mail-7.4.2-2022.02.23-000131/_search?size=0
</code></pre></div>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"aggs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"max_date"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"max"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"field"</span><span class="p">:</span><span class="w"> </span><span class="s2">"@timestamp"</span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"min_date"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"min"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"field"</span><span class="p">:</span><span class="w"> </span><span class="s2">"@timestamp"</span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>And then updating the index settings:</p>
<div class="codehilite"><pre><span></span><code>PUT mail-7.4.2-2022.02.23-000131/_settings
</code></pre></div>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"settings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"lifecycle.origination_date"</span><span class="p">:</span><span class="w"> </span><span class="mi">1646629200000</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Finally, check the ILM information:</p>
<div class="codehilite"><pre><span></span><code>GET mail-7.4.2-2022.02.23-000131/_ilm/explain
</code></pre></div>
<p>and verify that the age is as expected:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"indices"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"mail-7.4.2-2022.02.23-000131"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"mail-7.4.2-2022.02.23-000131"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"managed"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"policy"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"mail"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"lifecycle_date_millis"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1646629200000</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"age"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"28.25d"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"phase"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"hot"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"phase_time_millis"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1649003776346</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"action"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"complete"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"action_time_millis"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1649004257477</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"step"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"complete"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"step_time_millis"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1649004257477</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"phase_execution"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"policy"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"mail"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"phase_definition"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"min_age"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0ms"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"actions"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"rollover"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"max_size"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"50gb"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"max_age"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"30d"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"version"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"modified_date_in_millis"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">1647091493509</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>And you're set.</p>
License to talk upgraded2022-03-17T06:40:00-04:002022-03-17T06:40:00-04:00Gaige B. Paulsentag:www.gaige.net,2022-03-17:/license-to-talk-upgraded.html<p>After nearly 20 years with a Technician class ham license, I've finally taken (and passed)
the test to upgrade my license to General class. Next step is to try for my Extra class
in an attempt to upgrade before my 20th anniversary as a ham next year.</p>
First look 2021 M1 MacBook Pro2021-11-05T18:00:00-04:002021-11-05T18:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-11-05:/first-look-2021-m1-macbook-pro.html<p>I last bought a MacBook Pro from Apple in November of 2019, in the midst of a bout of
travel that was about to come to an end. In point of fact, I haven't used my trusty
MacBook Pro much in the last 19 months, since the COVID-19 pandemic started …</p><p>I last bought a MacBook Pro from Apple in November of 2019, in the midst of a bout of
travel that was about to come to an end. In point of fact, I haven't used my trusty
MacBook Pro much in the last 19 months, since the COVID-19 pandemic started, but it
still is my go-to machine for running to the hosting center, taking with on the few
trips we've been on, or just hanging out with Carol on the weekends. One thing that I
will say about it is that it's been a hot, and relatively noisy machine, and has led
me to keep a power cord in the front parlor now.</p>
<p>This is all coming to an end, as Apple "finally" released their professional M1 laptops,
and they look awesome. So far, I've only had mine for the weekend, so this is likely
to just be a first look at how it's working so far.</p>
<h1>M1 migration</h1>
<p>As I've written about before, I generally make a couple of full backups (one to
TimeMachine, one using CarbonCopyCloner) before I migrate to a new Mac using Apple's
Migration Assistant, and this time was no different than previous. Given that both the
2019 and 2021 MacBook Pros have Thunderbolt 3, I was able to hook them up over that
connection for some super-fast copying—after one false start.</p>
<p>Turns out that if you want to hook up two Macs over TB3 for Migration Assistant, you
want to have that connection in place before starting Migration Assistant on either of
them. Apple's typically smart at looking at what networking is available, and will choose
the TB3 direct connection over wired Ethernet or WiFI, but it appears to be the case
only if that's seen relatively early in the process. My first time through with these
two machines resulted in an 18-hour estimate based on running over WiFi, so I successfully
aborted the process and restarted it after getting both machines connected via TB3.</p>
<p>The process went off without a hitch and everything expected moved over. As is usually the
case nowadays, authorizations need to be reauthorized for new hardware, but other than
that and a couple of pieces of software that needed to be reactivated, it was a
straightforward process.</p>
<h1>Developer Performance</h1>
<p>I haven't done as thorough of a breakdown as I did with
<a href="https://www.gaige.net/developing-on-a-2019-mac-pro.html">Developing on a 2019 Mac Pro</a>, but I can report
that the preliminary results on Cartographica are excellent. Previously, the
2019 MacBook Pro took about twice as long to build and about 12% less time to run tests
than the Mac Pro. The test times have flattened out between the Mac Pro and the Intel
MacBook Pro, and the Mac Pro still builds in about half the time.
This is owing to some changes in my test jig, expansion of the software during the 1.5
rollout and overall improvement in test coverage.</p>
<p>The Cartographica results for the M1 MacBook Pro are extremely encouraging:</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th style="text-align:right">Build Time</th>
<th style="text-align:right">Test Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>2021 MacBook Pro (8+2 core M1 Max, 64GB)</td>
<td style="text-align:right">101s</td>
<td style="text-align:right">157s</td>
</tr>
<tr>
<td>2019 Mac Pro (16 core Xeon, 96GB)</td>
<td style="text-align:right">107s</td>
<td style="text-align:right">167s</td>
</tr>
<tr>
<td>2020 Mac Mini (4+4 core M1, 16GB)</td>
<td style="text-align:right">145s</td>
<td style="text-align:right">191s</td>
</tr>
<tr>
<td>2019 MacBook Pro (8 core i9, 32GB)</td>
<td style="text-align:right">202s</td>
<td style="text-align:right">167s</td>
</tr>
<tr>
<td>2018 Mac Mini (6 core i7, 32GB)</td>
<td style="text-align:right">251s</td>
<td style="text-align:right">178s</td>
</tr>
</tbody>
</table>
<p>Tests built against 128c43ef using Xcode 13.1.</p>
<p>It's important to note that the current version of Cartographica is not M1 native.
Unfortunately, a reasonably popular raster format has a third-party library that I
have not been able to isolate yet and is not available for Apple Silicon for the Mac.
This means that all the tests run in emulation on any non-Intel mac.</p>
<p>Also, interesting to note that the 2021 MacBook Pro ran the whole build/test
process with no fans and was still at 100% battery when it finished. I ran the
tests for the 2019 MacBook Pro while plugged in. Carol could hear the fans 7ft
away. Running on battery didn't substantially change either the performance or
the noise.</p>
<h1>Software compatibility</h1>
<p>So far, most of what I use on my MacBook Pro has been working fine. I did have some
old drivers for my Logitech MX Ergo trackball that didn't run correctly. Installing the
latest version of Logitech Options fixed that problem.</p>
<h2>Docker</h2>
<p>Docker was a bit of a surprise. I guess I hadn't been paying attention, so I didn't
expect it to automatically use QEMU to run an Intel VM in order to run my local DNS
server (the only long-running Docker container that I use). I haven't played around
much with ARM-based containers, but I now have the perfect place to build them.</p>
<h2>Brew</h2>
<p>Brew was great. I'd done some work with it when I put the M1 in the build farm, so
I was aware that the Intel and AS versions run in separate directory trees, making
it easy to install partially-Intel and partially-AS. At this point, I have only a couple
of items that are on intel and everything else is compiled for, and bottled for
Apple Silicon. There are good directions on the <a href="https://brew.sh">brew</a> site and
other places on the web.</p>
<h1>Battery performance</h1>
<p>So far, the battery performance has been crazy-good. I ran on laptop power for about
3 hours during one stretch this weekend and that resulted in an 13% drop in battery. It
wasn't the heaviest use I could do, but it was representative (I watched a bit of video,
surfed the net, and did some compilation and testing).</p>
<p>Following up on this, I let the battery run down for 3 days and then sat in the Parlor,
without battery and installed/removed software from brew and recompiled and debugged
Cartographica. That got me almost 4 hours of run time with intense workloads and heavy
networking (as well as a bunch of background processing for reindexing, etc.)</p>
<h1>Fans and lap comfort</h1>
<p>By my recconing, it's still shorts weather (it was 69°F yesterday), so the lap
experience is without the benefit of jeans. As such, I can happily report that you
might find the MacBook Pro does provide a little warming on cold days, it does not
scorch your legs like it's i9 predecessor did.</p>
<p>As for fans, maybe I'll write a follow-up to this when I notice them, but so far if
they're running, I couldn't tell you.</p>
<h1>Conclusion</h1>
<p>It's just a first look, but wow is this an interesting machine! So far, software and
hardware have worked great; I'm mostly getting used to the changes in macOS Monterey and
have basically forgotten about the <em>notch</em> (although I've been a long-time user of
<a href="https://www.macbartender.com/">Bartender</a> and I did move to using the Bartender Bar
again on the MacBook Pro to avoid collisions with my many menu bar items).</p>
<p>This machine is an amazing first-step into professional Apple Silicon, and based on it,
I fully expect that whatever they replace my 2019 Intel MacPro with will be an absolute
beast, an likely in ways that I won't find necessary.</p>
<p>My next question is: for my needs, is the 16" really necessary? I've bought the largest
laptop that Apple makes since the
<a href="https://en.wikipedia.org/wiki/Macintosh_Portable">1989 Macintosh Portable</a> and although
I'm very happy that I no longer carry a 16-lb beast with 1MB of RAM and a 16MHz 68000, the
question remains whether the additional screen real estate is really worth the tray table
issues on airplanes and the extra weight in my backpack. Previously, the 16" (and the 15"
before that, and the 17" before that) provided improved thermals, higher-spec'd processors,
and frequently better options for discrete graphics. All of these reasons appear to be
gone (save the "high power" mode that only exists on the 16"), so the question for me
is going to be: does the additional screen real estate counter the weight and seatback tray
compatibility? It remains to be seen.</p>
Bacula pruning2021-10-05T07:22:00-04:002021-10-05T07:22:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-10-05:/bacula-pruning.html<p>After 18 months of using Bacula and sending copies of my data to the cloud (in this
case, cloud I operate in another location) using an S3-compatible storage mechanism,
I noticed I had a lot of data sitting around on my current server for backups. When
I set out to …</p><p>After 18 months of using Bacula and sending copies of my data to the cloud (in this
case, cloud I operate in another location) using an S3-compatible storage mechanism,
I noticed I had a lot of data sitting around on my current server for backups. When
I set out to move to Bacula, I decided to use long retention times for my core
monthly full backups, which resulted in more than a small number of terrabytes of data.</p>
<p>At the time of the implementation (and still the case at the time of this writing), the
automatic options in Bacula for pruning/truncating local copies of cloud datasets were:</p>
<ul>
<li><em>No</em> (do not remove cache)</li>
<li><em>AfterUpload</em> (each part removed directly after upload)</li>
<li><em>AtEndOfJob</em> (each part removed at the end of the job)</li>
</ul>
<p>None of these would work for me, as I want to retain the data for months locally, only
giving up my cached copy when I'm outside of my normal restore window, or when I need the
space.</p>
<p>There are a number of ways to prune, depending on how much you want to get into the
Bacula mindset.</p>
<h3>Manual purge using find</h3>
<p>It turns out that if you leave the label intact (the label being <code>part.1</code>
in the volume directory), you can delete any parts in the cloud volume and they will
be auto-retrieved during a restore. This will allow you to override any settings you
have in <code>bacula-dir.conf</code> for your <code>CacheRetention</code> and just manually purge in any
way you like. In my case, I made use of find:</p>
<div class="codehilite"><pre><span></span><code>find<span class="w"> </span>.<span class="w"> </span>-regextype<span class="w"> </span>posix-egrep<span class="w"> </span>-regex<span class="w"> </span><span class="s1">'.*\/Vol-.*\/part\.([2-9]|..+)'</span><span class="w"> </span>-exec<span class="w"> </span>rm<span class="w"> </span><span class="se">\{\}</span><span class="w"> </span><span class="se">\;</span>
</code></pre></div>
<p>This particular command uses a posix regular expression to find any file in any directory
starting <code>Vol-</code> and named <code>part._number_</code> where <em>number</em> is any value other than 1.</p>
<h3>Manual pruning using bconsole</h3>
<p>Bacula's console (bconsole) has a Cloud command which can be used to force a
prune operation. The <code>cloud prune</code> command respects the <code>CacheRetention</code> setting and
has a number of command-line parameters to allow you to specify what you want to prune.
You can prune by <code>storage</code>, <code>pool</code>, or even <code>MediaType</code>. There is also a parameter
to prune <code>AllPools</code>.</p>
<p>In my case, I used:</p>
<div class="codehilite"><pre><span></span><code>cloud prune AllFromPool Storage=Cloud-CT Pool=File
</code></pre></div>
<p>which breaks down to:</p>
<ul>
<li><code>cloud</code> command</li>
<li><code>prune</code> sub-command</li>
<li><code>AllFromPool</code>: run the purge command on all volumes in the pool</li>
<li><code>Storage=</code>: use the specific Storage definition (in this case <code>Cloud-CT</code>)</li>
<li><code>Pool=</code>: use the specific Pool (in this case <code>File</code>)</li>
</ul>
<p>For ClueTrust, we use 3 different pools in our storage:</p>
<ul>
<li><code>File</code> for the full backups (historical naming convention)</li>
<li><code>Inc-File</code> for the daily incremental backups (from the last File backup)</li>
<li><code>Diff-File</code> for the weekly differential backups (from the last File backup)</li>
</ul>
<p>In this case, I only want to purge the full backups that are outside of the range of the
incremental and differential backups. To that end, I've set the <code>CacheRetention</code>
appropriately in my <code>bacula-dir.conf</code> file and so I can trust bacula to clear these
correctly.</p>
<h3>Automatic pruning using bacula admin jobs</h3>
<p>I've read that this is possible, but I haven't found the appropriate documentation yet.
At this point, I can't recommend, but the other two processes work fine and are easily
scripted if need be.</p>
Rclone to the rescue2021-10-05T06:01:00-04:002021-10-05T06:01:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-10-05:/rclone-to-the-rescue.html<p>Back in September of last year, I wrote in
<a href="https://www.gaige.net/bacula-6-months-on.html">Bacula: 6 months on</a>
that cloud backups required <code>part.0</code> in order to be recognized for automatic part retrieval.</p>
<p>While this was mostly accurate, the critical file is actually <code>part.1</code>. As such, when
referencing my own blog post when trimming …</p><p>Back in September of last year, I wrote in
<a href="https://www.gaige.net/bacula-6-months-on.html">Bacula: 6 months on</a>
that cloud backups required <code>part.0</code> in order to be recognized for automatic part retrieval.</p>
<p>While this was mostly accurate, the critical file is actually <code>part.1</code>. As such, when
referencing my own blog post when trimming my bacula storage, I deleted the wrong file,
leaving my "volumes" without labels, and thus rendering automatic part retrieval
inoperative.</p>
<p>To remedy this, I needed to sync the <code>part.1</code> files back into my local cache. As I'm
using an S3-style storage mechanism for my remotes, I decided to use
<a href="https://rclone.org">Rclone</a> to bring
the files back.</p>
<p>The command looks like this (this is the safe version that doesn't copy, hence <code>-n</code>,
remove that when you're satisfied it's good to go):</p>
<div class="codehilite"><pre><span></span><code>rclone<span class="w"> </span>sync<span class="w"> </span>-n<span class="w"> </span>--no-update-modtime<span class="w"> </span>s3-server-ref:s3-bucket<span class="w"> </span><span class="nb">local</span><span class="w"> </span>--include<span class="w"> </span><span class="s2">"part.1"</span>
</code></pre></div>
<p>Breaking this command down:</p>
<ul>
<li>Use the <code>rclone</code> command</li>
<li><code>sync</code> subcommand will synchronize from source to destination</li>
<li><code>--no-update-modtime</code> attempts to leave the modification time the same</li>
<li><code>s3-server-ref</code> is a reference to an Rclone server created with <code>rclone config</code></li>
<li><code>s3-bucket</code> is the bucket (or path) of the source</li>
<li><code>local</code> is the path to the local destination</li>
<li><code>--include "part.1"</code> is a filter that only copies the specific filename</li>
</ul>
<p>Running the version as a dry run (<code>-n</code>) results in (partial results):</p>
<div class="codehilite"><pre><span></span><code><span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-0998/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
<span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-1101/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
<span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-1105/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
<span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-1106/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
<span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-1110/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
<span class="m">2021</span>/10/05<span class="w"> </span><span class="m">10</span>:06:56<span class="w"> </span>NOTICE:<span class="w"> </span>Vol-0999/part.1:<span class="w"> </span>Not<span class="w"> </span>copying<span class="w"> </span>as<span class="w"> </span>--dry-run
</code></pre></div>
<p>which indicates that each of these files would be copied if this had not been a dry run.</p>
<p>Re-run the command removing <code>-n</code> and you should get the desired results.</p>
GitLab stuck MR2021-09-26T17:40:00-04:002021-09-26T17:40:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-09-26:/gitlab-stuck-mr.html<p>MRs (Merge Requests) in GitLab are similar to PRs (Pull Requests) in GitHub, although
the process and language around them are slightly different. The name specifically
refers the the request to merge into another branch from a branch (or repository) that
isn't the same. Simple enough.</p>
<p>Most of the time …</p><p>MRs (Merge Requests) in GitLab are similar to PRs (Pull Requests) in GitHub, although
the process and language around them are slightly different. The name specifically
refers the the request to merge into another branch from a branch (or repository) that
isn't the same. Simple enough.</p>
<p>Most of the time since moving to GitLab I've been extremely happy with both our productivity
and the stability of GitLab, especially considering that we're running it from a docker
container in SmartOS. All told, it's been a really good experience.</p>
<p>With that said, it hasn't come without bumps. Mostly these are related to the operating
environment, and result in relatively straightforward failures, such as an inability
to see memory consumption correctly.</p>
<p>Today, though, I got a bonus problem. This one took some real time and resulted in not
only digging through gitlab.com's issues, but dropping into the rails console for gitlab
to resolve it.</p>
<p>I'm not sure what actually caused this problem, and it may well be that I did or am
doing something problematic. But, the result was that I had a Merge Request (MR) that
was in the "merging" state for nearly an hour. Considering most other merges have taken
seconds, even with my large codebase for Cartographica, it seemed like something was wrong.</p>
<p>I dug around on <a href="https://gitlab.com">gitlab.com</a> looking for some answers and ran across a couple
of aging examples of a similar "stuck" MR.</p>
<p>If you find yourself in this position, you may want to look at:</p>
<p><a href="https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18048">Issue 18048</a> which details a
few different diagnostics and work-arounds.</p>
<p>I used the gitlab console by logging in to the server container and</p>
<div class="codehilite"><pre><span></span><code>gitlab-rails<span class="w"> </span>console
</code></pre></div>
<p>to get the console running and then:</p>
<div class="codehilite"><pre><span></span><code><span class="n">proj</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="no">Project</span><span class="o">.</span><span class="n">find_by_full_path</span><span class="p">(</span><span class="s1">'namespace/myproject'</span><span class="p">)</span>
<span class="n">mr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">proj</span><span class="o">.</span><span class="n">merge_requests</span><span class="o">.</span><span class="n">find_by</span><span class="p">(</span><span class="ss">iid</span><span class="p">:</span><span class="w"> </span><span class="mi">123</span><span class="p">)</span>
<span class="n">mr</span><span class="o">.</span><span class="n">state</span>
</code></pre></div>
<p>This prints out the status of the merge request, which in my case was <code>locked</code>.</p>
<p>I did the following:</p>
<ol>
<li>Verified that the record looked OK by using <code>mr.valid?</code></li>
<li>Unlocked the MR using <code>mr.unlock_mr</code>, which resulted in <code>true</code></li>
</ol>
<p>After that, the state returned to <code>open</code> and I was able to go to the UI and merge it.</p>
<p>Further down there were recommendations for going all the way to the database console
using:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># login to prostgres instance</span>
sudo<span class="w"> </span>gitlab-rails<span class="w"> </span>dbconsole
--<span class="w"> </span>check<span class="w"> </span>the<span class="w"> </span>list<span class="w"> </span>of<span class="w"> </span>locked<span class="w"> </span>merge<span class="w"> </span>requests
<span class="k">select</span><span class="w"> </span>id,<span class="w"> </span>iid<span class="w"> </span>as<span class="w"> </span>merge_id,<span class="w"> </span>source_branch,<span class="w"> </span>target_branch,<span class="w"> </span>locked_at,<span class="w"> </span>merge_error,<span class="w"> </span>merge_commit_sha,<span class="w"> </span>in_progress_merge_commit_sha,<span class="w"> </span>merge_status,<span class="w"> </span>state<span class="w"> </span>from<span class="w"> </span>merge_requests<span class="w"> </span>where<span class="w"> </span><span class="nv">state</span><span class="o">=</span><span class="s1">'locked'</span><span class="w"> </span>order<span class="w"> </span>by<span class="w"> </span>id<span class="w"> </span>desc<span class="p">;</span>
</code></pre></div>
<p>But, I didn't find that necessary.</p>
<p>I'll put here (lest I forget it later) that I also made use of the log tailing mechanism
(<code>gitlab-ctl tail</code>) which can also be directed a specific log by adding an argument.</p>
Pelican plugin updates2021-08-29T07:44:00-04:002021-08-29T07:44:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-08-29:/pelican-plugin-updates.html<p>One of the advantages of our recent pivot to gitlab is that I'm spending some time
looking at existing repositories and doing some updates.</p>
<p>Most of my repos are private and hosted on our private gitlab server. For public code,
I generally place it on <a href="https://github.com">GitHub</a>.
With the recent automation …</p><p>One of the advantages of our recent pivot to gitlab is that I'm spending some time
looking at existing repositories and doing some updates.</p>
<p>Most of my repos are private and hosted on our private gitlab server. For public code,
I generally place it on <a href="https://github.com">GitHub</a>.
With the recent automation changes on GitHub, I've been doing some updates to get CI
running there as well. For the most part, this uses whatever templates exist for the
project (in the case of plugins, for example) or the GitHub default.</p>
<h2>Updating pelican plugins</h2>
<p>I've mentioned before my move to
<a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">static site generation</a>,
and also the plugins that I've created or modified for Pelican:</p>
<ul>
<li><a href="https://www.gaige.net/update-to-nginx_alias_map.html">nginx_alias_map</a></li>
<li><a href="https://www.gaige.net/pre-commit-and-pelican.html">markdown-it-reader</a></li>
</ul>
<p>Yesterday, I decided to do a little clean-up of both of these plugins. In particular:</p>
<ol>
<li>Updated <code>nginx_alias_map</code> to use the Pelican cookiecutter templated</li>
<li>Added tests to both plugins</li>
<li>Verified that CI was working correctly for both plugins</li>
<li>Fixed a Python 3.6-specific bug in <code>nginx_alias_map</code></li>
<li>Bumped versions to 1.0/Production and changed environment to <code>Plugins</code></li>
</ol>
<p>The big change here is that <code>nginx_alias_map</code> is now on <a href="https://pypi.org">PyPI</a>
and can now be installed as a dependency without having to use a git
submodule as I'd been doing previously.</p>
<p>If you're interested, check out:</p>
<ul>
<li><a href="https://pypi.org/project/pelican-nginx-alias-map/">nginx_alias_map</a></li>
<li><a href="https://pypi.org/project/pelican-markdown-it-reader/">markdown-it-reader</a></li>
</ul>
<p>They're both easier to use now, with <code>pip install pelican-nginx-alias-map</code> or
<code>pip install pelican-markdown-it-reader</code> as the installation method.</p>
<p>To support using <code>pre-commit</code> with pelican (as detailed in
<a href="https://www.gaige.net/pre-commit-and-pelican.html">Pre-commit and Pelican</a>), I have
also updated my <code>mdformat</code> plugins:</p>
<ul>
<li><a href="https://github.com/gaige/mdformat-footnote">mdformat-footnote</a></li>
<li><a href="https://github.com/gaige/mdformat-pelican">mdformat-pelican</a></li>
</ul>
<p>These (respectively) avoid reformatting or errors when using the footnote plugin
with markdown-it, or when using pelican-specific items, such as
<code>{filename}</code>.</p>
Deploying with Gitlab2021-07-25T18:49:00-04:002021-07-25T18:49:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-07-25:/deploying-with-gitlab.html<p>In June, I mentioned in an article about <a href="https://www.gaige.net/docker-on-smartos.html">Docker on SmartOS</a>
that we are doing some work with <a href="https://gitlab.com">GitLab</a>
these days as a replacement for my venerable <a href="https://gitolite.com">Gitolite</a> server (and, to an increasing
extent <a href="https://jenkins.io">Jenkins</a>).</p>
<h2>Deploying from Pelican</h2>
<p>I'm likely going to write more on GitLab in the near future …</p><p>In June, I mentioned in an article about <a href="https://www.gaige.net/docker-on-smartos.html">Docker on SmartOS</a>
that we are doing some work with <a href="https://gitlab.com">GitLab</a>
these days as a replacement for my venerable <a href="https://gitolite.com">Gitolite</a> server (and, to an increasing
extent <a href="https://jenkins.io">Jenkins</a>).</p>
<h2>Deploying from Pelican</h2>
<p>I'm likely going to write more on GitLab in the near future, but for now, I'd
like to document some things I've learned about deploying with Gitlab.</p>
<p>This blog is deployed in a semi-automated fashion. As mentioned
<a href="https://www.gaige.net/static-pages-18-months-on.html">previously</a>,
it is compiled using <a href="https://blog.getpelican.com"><code>pelican</code></a>
and served as static pages using <a href="http://nginx.org">nginx</a>.</p>
<p>As such, once modifications are made, I'm ready to verify that they look OK
and work correctly on the Stage Server; once I'm happy with that deployment,
it's time to push to production.</p>
<p>Historically, I started out by doing a complete rebuild of the server serving
up the pages. THat got tedious if I was writing a lot of posts (or, at least
if I was writing posts more frequently than OS releases and nginx releases).
Eventually, I modified my <a href="https://www.ansible.com">ansible</a> scripts so that
they had a tag for <code>publish</code> which would skip the re-provisioning process
and the process of building new certs, etc. and just deploy the latest pelican,
build the pages, and reset the cache. In fact, it would do so in a separate
directory, so that it would flash-cut the web pages.</p>
<p>While rolling out GitLab, I started playing with the CI tools and realized there
was a lot I could do with it, much of it more easily than I could with Jenkins.
As such, an automatic build to stage followed by a manually-triggered build to
production was simple to configure.</p>
<p>So, I set out on my next automation journey with GitLab...</p>
<h2>Access control</h2>
<p>One nice thing about running the CI under the rubric of the SCM is that you can
grant permissions to do source-related things just from the SCM. This makes it
simple to pull from multiple repositories and perform other SCM-specific tasks.</p>
<p>However, this doesn't specifically extend beyond the CI and SCM and into the
deployment. So, my next question was how to control access to the hosts and
make sure that I could control them, and retrieve the code without trouble.</p>
<p>Further, I wanted to re-use the ansible playbooks that I used to deploy the
systems (albeit with tags to reduce the plays), while limiting access to the
stage and production servers (not the SmartOS global zones they're deployed
from). Since I was reusing these mechanisms, I wanted to leave the existing
ssh-based access controls in place.</p>
<p>As an aside, I could now switch my deployment method for git repositories to
using deployment or personal access tokens, but I'd rather not right now.</p>
<h2>SSH solution</h2>
<p>My existing deployment pattern automatically deals with what I refer to as
<code>ssh_access_keys</code>, which are SSH keys that are used for root access to the
servers. These are generally used infrequently (there are separate deployment
keys that are multi-server), but when accessing only the VM, the
<code>ssh_access_keys</code> are precisely the right tool.</p>
<p>When running on the CI server, I need to adopt the ssh key as part of the
CI process, and I use <code>ssh-agent</code> to do that (one agent per running CI
process, segregated by the socket/pid combination). It's simple to start this by
using:</p>
<div class="codehilite"><pre><span></span><code>eval $(ssh-agent -s)
</code></pre></div>
<p>This creates the agent and sets the shell variables so the agent is reachable.</p>
<p>Then comes the real trick: loading the ssh key into the agent. I had a vague
recollection that it was possible to load a key from a shell variable, and
here's how to do it:</p>
<div class="codehilite"><pre><span></span><code>echo "$HOST_DEPLOY_KEY" | tr -d '\r' | ssh-add -
</code></pre></div>
<h2>Ansible ssh control paths</h2>
<p>While getting this put together, I ran across an issue with the length of the
path for <code>ANSIBLE_SSH_CONTROL_PATH</code>, which is used by SSH to persist connections
(in our configuration). Especially on Solaris (and derivatives, like SmartOS),
there's a path limit to the control file and caused a problem with the
relatively-deep nesting that gitlab runners do for their paths. The solution
was to define a bespoke path:</p>
<div class="codehilite"><pre><span></span><code><span class="k">export</span><span class="w"> </span><span class="n">ANSIBLE_SSH_CONTROL_PATH_DIR</span><span class="o">=/</span><span class="n">tmp</span><span class="o">/$</span><span class="p">{</span><span class="n">CI_JOB_ID</span><span class="p">}</span><span class="o">-$</span><span class="p">{</span><span class="n">CI_COMMIT_SHORT_SHA</span><span class="p">}</span><span class="o">/.</span><span class="n">ansible</span><span class="o">/</span><span class="n">cp</span>
</code></pre></div>
<p>Not that this path is in <code>/tmp</code>, not in <code>~</code> and certainly not in the build directory,
however, it does change for every job and repo.</p>
<h2>Final gitlab script</h2>
<div class="codehilite"><pre><span></span><code><span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eval $(ssh-agent -s)</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo "$HOST_DEPLOY_KEY" | tr -d '\r' | ssh-add -</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">export HOME=$(pwd)</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">export ANSIBLE_SSH_CONTROL_PATH_DIR=/tmp/${CI_JOB_ID}-${CI_COMMIT_SHORT_SHA}/.ansible/cp</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'git</span><span class="nv"> </span><span class="s">config</span><span class="nv"> </span><span class="s">--global</span><span class="nv"> </span><span class="s">url."https://gitlab-ci-token:${CI_JOB_TOKEN}@your.git.server/".insteadOf</span><span class="nv"> </span><span class="s">git@your.git.server:'</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@your.git.server/playbooks/ansible-web.git</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cd ansible-web</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ansible-galaxy install -r requirements.yml -f</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ansible-playbook -i stage -t publish --vault-password-file $VAULT_SECRET -e cert_renew_days=0 pelican.yml</span>
</code></pre></div>
<p>Putting it all together:</p>
<ul>
<li>set up SSH (lines 1 & 2)</li>
<li>set HOME so that we're not stomping on another cache; this may not be necessary
if you can guarantee that only one runner will be running at a time in each
account (line 3)</li>
<li>set the ansible control path (line 4)</li>
<li>rewrite our git URLs globally (line 5)</li>
<li>check out our ansible playbook repository (line 6)</li>
<li>update the ansible galaxy requirements (lines 7 & 8)</li>
<li>Run the playbook to our stage sever (line 9)</li>
</ul>
<p>You might be wondering about line 5, where we use an interesting feature of git
to rewrite the URLs. This might not be explicitly necessary if I were to allow
the ssh key that I use for deployment access to all of my dependencies in my
git repo. However, I've left it that way for future compatibility and because
it confines this particular script to being run by the CI server.</p>
<p>So, for those keeping score, the gitlab server runs the gitlab script on a SmartOS host
which runs the gitlab agent, and thus the ansible runs on SmartOS.
Theoretically the ansible could run on basically anything (my Jenkins versions of
this ran on macOS Jenkins nodes), but our provisioning is done from SmartOS these days,
so keeping things the same is a good thing.</p>
<h2>Manually-triggered releases</h2>
<p>I mentioned in the beginning that I was going to be manually trigging the release
to production. This is done using a rule in the GitLab CI configuration:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">deploy-prod</span><span class="p">:</span>
<span class="w"> </span><span class="nt">tags</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">ansible</span><span class="p p-Indicator">]</span>
<span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">prod</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">production</span>
<span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://${SERVER}</span>
<span class="w"> </span><span class="nt">script</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"...</span><span class="nv"> </span><span class="s">see</span><span class="nv"> </span><span class="s">above</span><span class="nv"> </span><span class="s">..."</span>
<span class="w"> </span><span class="nt">rules</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span>
<span class="w"> </span><span class="nt">when</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">manual</span>
</code></pre></div>
<p>This job definition requires that the runner be tagged <code>ansible</code>, names the stage
<code>prod</code>, sets up an environment for prod with the URL pointing at our final
server, includes the script above, and then conditionally (and only on the
main branch) holds for manual release.</p>
<h2>Script locations</h2>
<p>One additional note I'll make is that I made some potentially interesting
decisions on where to place the gitlab scripts. Since I tend to have multiple
hosts (or groups) using the same ansible plays, I knew I wanted
a place to share the scripts calling them. Their requirements tend to be more
aligned with the ansible playbooks than the code that is deployed. As such, I
placed the gitlab-ci jobs as templates in my ansible playbook repositories in
a <code>gitlab-deploy</code> directory. I aligned the names with the playbooks.</p>
<p>To call these, I use the <code>include</code> directive in the <code>.gitlab-ci.yml</code> files for
the repositories I'm deploying:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># This will work, but not on the python runner (yet)</span>
<span class="nt">include</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">project</span><span class="p">:</span><span class="w"> </span><span class="s">'playbooks/ansible-web'</span>
<span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="s">'gitlab-deploy/pelican.yml'</span>
<span class="nt">variables</span><span class="p">:</span>
<span class="w"> </span><span class="nt">SERVER_GROUP</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gaiges_pages</span>
<span class="w"> </span><span class="nt">SERVER</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">www.gaige.net</span>
</code></pre></div>
<p>Note the additional variables. Since this deployment script is used by both
the <a href="https://gaige.net">Gaige's Pages</a> and <a href="https://blog.cartographica.com">Cartographica</a>
blogs, I needed a way to pass in the server and group names.</p>
Docker on SmartOS2021-06-11T10:00:00-04:002021-06-11T10:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-06-11:/docker-on-smartos.html<p>This spring, there was a some movement on the Illumos/SmartOS front in
implementing features to better support running LX zones with Linux variants.
Since Docker images (generally) run on Linux underpinnings, support for running
Docker images on SmartOS are dependent upon this support working correctly.</p>
<p>For those familiar with …</p><p>This spring, there was a some movement on the Illumos/SmartOS front in
implementing features to better support running LX zones with Linux variants.
Since Docker images (generally) run on Linux underpinnings, support for running
Docker images on SmartOS are dependent upon this support working correctly.</p>
<p>For those familiar with <a href="https://www.joyent.com/triton/on-premise">Triton</a>, you
know that Triton can run Docker directly as part of its standard configuration. But,
for those of us who don't run Triton, but do run SmartOS, there are some steps that can
be taken to use Docker images under SmartOS.</p>
<p>To provide a concrete example, I'm going to use GitLab as the example in this article.</p>
<h2>Docker on SmartOS, the harder way</h2>
<p>I'm completely stealing that line (and adapting some of the content) from a
<a href="https://jasper.la/posts/docker-on-smartos-the-harder-way/">2016 blog post</a>
by Jasper Lievisse Adriaanse of the same name.</p>
<p>His summary of using Docker on SmartOS was the best resource that I found for
creating virtual machines using <code>vmadm</code>.</p>
<h3>Getting docker images on SmartOS</h3>
<p><a href="https://hub.docker.com">Docker Hub</a> is "the world's largest library... for container images"
and as such is basically where you want to go to get your docker images.</p>
<p>To get access to Docker Hub, you need to add the source to <code>imgadm</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># imgadm sources --add-docker-hub</span>
</code></pre></div>
<p>As noted in the post, <code>imgadm avail</code> doesn't work against Docker Hub, so you'll need
to search there manually or get it directly from another source. Once you know
what docker image you need, you can add it using <code>imgadm import</code>.</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># imgadm import gitlab/gitlab-ee:latest</span>
</code></pre></div>
<p>As is common with docker, some of the items in the image descriptor can be left
off, most notably <code>:latest</code> can be omitted and the <code>latest</code> tag will be used by
default.</p>
<p>Once you have the images loaded, you can see them weeded out from the rest of your images
by using:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># imgadm list --docker</span>
UUID<span class="w"> </span>REPOSITORY<span class="w"> </span>TAG<span class="w"> </span>IMAGE_ID<span class="w"> </span>CREATED
a001c571-b91a-a5b0-c251-0514a0d4a174<span class="w"> </span>gitlab/gitlab-ce<span class="w"> </span>latest<span class="w"> </span>sha256:42486<span class="w"> </span><span class="m">2021</span>-06-07T19:28:36Z
9080e799-d964-782e-e369-87d339e50798<span class="w"> </span>gitlab/gitlab-ee<span class="w"> </span>latest<span class="w"> </span>sha256:1f383<span class="w"> </span><span class="m">2021</span>-06-07T19:36:13Z
</code></pre></div>
<h3>Reasoning through the requirements</h3>
<p>One disadvantage of using SmartOS natively for docker in comparison to using docker
on Linux is that there isn't a docker control daemon to set things up for you. As
such, you'll need to dig in a bit to the requirements in order to make sure you
have all of the right sittings to get up and running.</p>
<p>You'll need to take a look at the docker parameters. Some of these parameters are baked
in to the images during the build phase and others are usually shown in command line
arguments in the instructions to run the code. As is frequently the case, there are
some of each to pay attention to in gitlab.</p>
<p>The <a href="https://docs.gitlab.com/omnibus/docker/">instructions for running gitlab in docker</a>
(as of 2021-06-11) call for the following docker command line:</p>
<div class="codehilite"><pre><span></span><code>sudo<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>--detach<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--hostname<span class="w"> </span>gitlab.example.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--publish<span class="w"> </span><span class="m">443</span>:443<span class="w"> </span>--publish<span class="w"> </span><span class="m">80</span>:80<span class="w"> </span>--publish<span class="w"> </span><span class="m">22</span>:22<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--name<span class="w"> </span>gitlab<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--restart<span class="w"> </span>always<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--volume<span class="w"> </span><span class="nv">$GITLAB_HOME</span>/config:/etc/gitlab<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--volume<span class="w"> </span><span class="nv">$GITLAB_HOME</span>/logs:/var/log/gitlab<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--volume<span class="w"> </span><span class="nv">$GITLAB_HOME</span>/data:/var/opt/gitlab<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>gitlab/gitlab-ee:latest
</code></pre></div>
<p>Let's look at the key parameters here:</p>
<table>
<thead>
<tr>
<th>argument</th>
<th>docker</th>
<th>SmartOS json</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>hostname</td>
<td>gitlab.example.com</td>
<td>hostname</td>
<td></td>
</tr>
<tr>
<td>publish</td>
<td>443:443, 80:80, 22:22</td>
<td>N/A</td>
<td>See network section</td>
</tr>
<tr>
<td>name</td>
<td>gitlab</td>
<td>alias</td>
<td>I used the FQDN here</td>
</tr>
<tr>
<td>restart</td>
<td>always</td>
<td>N/A</td>
<td>no equivalent</td>
</tr>
<tr>
<td>volume</td>
<td><em>various</em></td>
<td>filesystems</td>
<td>See file system section</td>
</tr>
</tbody>
</table>
<p>In addition to the parameters on the command line, there are also parameters inherent in
the docker container that we need to propagate to the SmartOS JSON.</p>
<p>We can see the key information in the JSON that comes with the docker images by using</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># imgadm info <uuid></span>
</code></pre></div>
<p>which will output the json for the image.</p>
<p>The key section to look for is the <code>tags</code> section, which in this version of gitlab-ee
contains:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"docker:repo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gitlab/gitlab-ee"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker:id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"sha256:1f38337b3401d2536562e4323999233b665aa41a2e6ef2c7509a0b938e53d94d"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker:architecture"</span><span class="p">:</span><span class="w"> </span><span class="s2">"amd64"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker:tag:latest"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker:config"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"Cmd"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"/assets/wrapper"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"Entrypoint"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"Env"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"LANG=C.UTF-8"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"TERM=xterm"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"WorkingDir"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span>
<span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>Again, we'll look at the key parameters of the <code>docker:config</code> object:</p>
<table>
<thead>
<tr>
<th>json path</th>
<th>value</th>
<th>SmartOS json</th>
<th>notes</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>Cmd</code></td>
<td><code>['/assets/wrapper']</code></td>
<td><code>docker:cmd</code></td>
<td></td>
</tr>
<tr>
<td><code>Entrypoint</code></td>
<td><code>null</code></td>
<td><code>docker:entrypoint</code></td>
<td>drop in this case, since it's empty</td>
</tr>
<tr>
<td><code>Env</code></td>
<td><code>[ "PATH=...", ... ]</code></td>
<td><code>docker:env</code></td>
<td>The entire JSON here needs to be encoded as a single string value</td>
</tr>
<tr>
<td><code>WorkingDir</code></td>
<td><code>""</code></td>
<td><code>docker:workdir</code></td>
<td></td>
</tr>
<tr>
<td><code>User</code></td>
<td><em>not present</em></td>
<td><code>docker:user</code></td>
<td>When commands need to run as a specific user</td>
</tr>
</tbody>
</table>
<p>It's important to note that the "dockery" configuration elements are all strings, so you
need to appropriately quote them to get them in the <code>internal_metadata</code> portion of your
json for <code>vmadm</code>.</p>
<h3>Setting up the storage</h3>
<p>Your specific mileage may vary. In many cases, images may run only with ephemeral storage,
in which case, you have nothing to do for <code>filesystems</code>, but in this example case, we have
three specific mount points: <code>/etc/gitlab</code>, <code>/var/log/gitlab</code>, and <code>/var/opt/gitlab</code>. In
our SmartOS systems, we use a separate <code>data</code> pool (usuallly spinning rust, in contrast
to the <code>zones</code> pool, which is all SSD), so you'll see that in the example. Further,
we have a naming standard for volumes that requires the FQDN and then the mount point.</p>
<p>Once you've figured out what zfs zones you need, create them using <code>zfs create</code>. I'll
leave that as an exercise for the reader.</p>
<h3>Constructing the vmadm json</h3>
<p>As is frequently the case (I may address doing this in ansible at a later date, but
so far these are all bespoke), you want to have a json file containing the parameters
of the new zone, so that you can pass them along using <code>vmadm create -f x.json</code>.</p>
<p>Now, we need to put together what we know from the information we've gathered so far:</p>
<ol>
<li>Start with a template LX zone</li>
<li>Make sure <code>brand</code> is <code>lx</code> and <code>docker</code> is <code>true</code></li>
<li>Set the docker image UUID in <code>image_uuid</code></li>
<li>Set up your network as required (see the <a href="#gitlab_network">gitlab note</a> below for
an understanding of why there are two interfaces)</li>
<li>Configure your file systems based on the required mount points</li>
<li>Put the operative docker information in the <code>internal_metadata</code> section</li>
<li>You will need to put an <code>owner_uuid</code> in the json because it is theoretically required
by the firewall code.</li>
</ol>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"alias"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gitlab.example.com"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"hostname"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gitlab.example.com"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"image_uuid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"9080e799-d964-782e-e369-87d339e50798"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"owner_uuid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"f834f98a-cac8-11eb-8ca3-cbddba9a698b"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"nics"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"nic_tag"</span><span class="p">:</span><span class="w"> </span><span class="s2">"vlan"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ips"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"X.X.X.X/24"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"gateways"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"X.X.X.1"</span><span class="p">],</span>
<span class="w"> </span><span class="nt">"vlan_id"</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"nic_tag"</span><span class="p">:</span><span class="w"> </span><span class="s2">"vlan"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"ips"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"Y.Y.Y.Y/24"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"vlan_id"</span><span class="p">:</span><span class="w"> </span><span class="mi">200</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"filesystems"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/data/gitlab.example.com/config"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"target"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/gitlab"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lofs"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/data/gitlab.example.com/logs"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"target"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/var/log/gitlab"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lofs"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/data/gitlab.example.com/data"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"target"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/var/opt/gitlab"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lofs"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"brand"</span><span class="p">:</span><span class="w"> </span><span class="s2">"lx"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"kernel_version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5.4.0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"max_physical_memory"</span><span class="p">:</span><span class="w"> </span><span class="mi">8192</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"maintain_resolvers"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"resolvers"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"8.8.8.8"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"quota"</span><span class="p">:</span><span class="mi">100</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"internal_metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"docker:cmd"</span><span class="p">:</span><span class="w"> </span><span class="s2">"[\"/assets/wrapper\"]"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"docker:env"</span><span class="w"> </span><span class="p">:</span>
<span class="w"> </span><span class="s2">"[ \"PATH=/opt/gitlab/embedded/bin:/opt/gitlab/bin:/assets:⏎</span>
<span class="s2"> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",⏎</span>
<span class="s2"> \"LANG=C.UTF-8\",⏎</span>
<span class="s2"> \"TERM=xterm\" ]"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p><strong>NOTE:</strong> I've split the <code>docker:env</code> line above for readibility. You'll need to keep
that as a single line for it to work correctly. Remember that all of the docker items are
strings, so they can't be split up. I've marked the returns for readibilty with <code>⏎</code> above.</p>
<p>Once this is prepared, you should be able to bring the zone up with</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># vmadm create -f myzone.json</span>
</code></pre></div>
<h3>Networking <em>or</em> Protecting your container from the internet</h3>
<p>One important thing to note about docker containers is that they all assume they're not
accessible from the internet. Under normal circumstances they're running on a loopback
or maybe an internal network, but certainly not on the internet. Depending on how your
systems are set up, this may be an issue, as <em>internal</em> services are not protected using
any normal process. You may get lucky and the folks who build the Docker container may
have used loopback for everything not going off-container, but don't count on it.</p>
<p>This protection is appropriate for SmartOS built-in firewall system, which is controlled
by <code>fwadm</code> from the global zone. Generally speaking, you want to firewall everything except the ports that
were specifically mentioned in the <code>publish</code> argument to the docker command.</p>
<p>This is where the <code>owner_id</code> comes in. It turns out this is necessary to run the firewall
in the global zone. No big deal, and you can add it later using <code>vmadm set</code> before you
enable the firewall if you forgot to do so prior to pulling the system up.</p>
<ol>
<li>
<p>Start the firewall for your container</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># fwadm start 85effadf-f4d3-63ca-cec4-8549b8797f75</span>
</code></pre></div>
</li>
<li>
<p>Enable access to your ports</p>
<div class="codehilite"><pre><span></span><code>fwadm<span class="w"> </span>add<span class="w"> </span>-e<span class="w"> </span>--desc<span class="w"> </span><span class="s1">'allow docker ports'</span><span class="w"> </span>-O<span class="w"> </span>5a2e68b0-ca8d-11eb-944f-8b6840c190dc⏎
<span class="w"> </span><span class="s2">"FROM any ⏎</span>
<span class="s2"> TO vm 85effadf-f4d3-63ca-cec4-8549b8797f75 ⏎</span>
<span class="s2"> ALLOW tcp (PORT 80 AND PORT 443 AND PORT 22)"</span>
</code></pre></div>
<p>(again, I'm using the counter-intuitive <code>⏎</code> to mean you <em>should not</em> put a return there)</p>
<p>Should be self-explanatory, but the value after <code>vm</code> is the current zone's UUID,
The value after <code>-O</code> is the owner UUID.</p>
</li>
</ol>
<h3>Debugging</h3>
<p>In the end, the vms here are just LX zones running on SmartOS, so you can still access
their filesystems and you can execute commands on them using <code>zlogin</code> (if you're careful).</p>
<ul>
<li>Look in <code>/zones/${UUID}/logs/stdio.log</code> for stdio of the docker environment</li>
<li>Start a shell using <code>zlogin -i ${UUID} /native/usr/vm/sbin/dockerexec /bin/sh</code></li>
<li>You can run an arbitrary command in the container using <code>zlogin -i ${UUID} /native/usr/vm/sbin/dockerexec</code>,
the <code>/bin/sh</code> is just a specifically useful example</li>
</ul>
<h3>Additional hints</h3>
<p>Here are some practical hints that I have developed by getting some of these images
(and others) running in SmartOS:</p>
<ul>
<li>
<p><a name="gitlab_network"></a>The gitlab binaries appear to really want a private interface in order to look for other
nodes in a potential cluster. As such, I found I needed to add a second, private
network interface so that it didn't get confused. This was made clear by error messages
thrown by the startup code.</p>
</li>
<li>
<p>Some Linux calls still don't work exactly the same in LX zones. This seems to be particularly
The case with process and system information gathering. In the case of gitlab, the CE
version ran with few changes to the configuration; but the EE version required reducing
the worker count to "0" (actually 1 worker, but that's the semaphore). This is another
place to look when debugging LX zones in general, and docker images in particular.</p>
</li>
</ul>
<h3>Additional links</h3>
<ul>
<li><a href="https://www.cyber-tec.org/2018/02/11/run-docker-images-on-smartos/">Run Docker images on SmartOS</a></li>
<li><a href="https://github.com/joyent/smartos-live/tree/master/src/dockerinit">dockerinit source</a></li>
</ul>
Pivoting Elasticsearch data2021-05-31T10:00:00-04:002021-05-31T10:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-05-31:/pivoting-elasticsearch-data.html<p>As I've possibly mentioned here before, ClueTrust is using Elasticsearch to
perform analysis of log information. Recently, I finally decided
to take some our telemetry inforamtion and pull it in to Elasticsearch as a
data exploration and statistical tool.</p>
<h2>Importing structured XML data into Elasticsearch</h2>
<p>Although there are some filters …</p><p>As I've possibly mentioned here before, ClueTrust is using Elasticsearch to
perform analysis of log information. Recently, I finally decided
to take some our telemetry inforamtion and pull it in to Elasticsearch as a
data exploration and statistical tool.</p>
<h2>Importing structured XML data into Elasticsearch</h2>
<p>Although there are some filters and logstash methods that have this capability,
the XML that we use is extremely regular (strict schemas, etc.), and I felt that
it would be better to directly and intentionally import based on the DOM that
I'd created in 2009 when preparing for Cartographica to ship.</p>
<p>For purposes of illustration, the basic form of the Cartographica telemetry
files is:</p>
<ul>
<li>Preamble</li>
<li>Crash logs (yep, they're embedded)</li>
<li>Event stream
<ul>
<li>Errors</li>
<li>Events (launch, quit, and other)</li>
<li>Exceptions</li>
<li>Statistics (at quit and other times)</li>
</ul>
</li>
</ul>
<p>Due to the way that Elasticsearch works, it turns out this is a really workable
input, generating a (possibly too verbose) set of items from each telemetry
report, including:</p>
<ul>
<li>Report</li>
<li>Launch</li>
<li>Event</li>
<li>Crash</li>
<li>Error</li>
<li>Exception</li>
<li>Statistic</li>
</ul>
<p>For most of these items, the format is regular and arguments are inserted
directly into the record (so a Crash has a crash log along with some interpretive
data as well as the preamble from the report). This holds true for Events and
Errors as basically individual data points inside of a preamble+launch context.
The only oddity is the Statistic report which contains many "columns" of data
for each statistic event. It's not lost on me that this idempotent data set is
very SNMP-like.</p>
<h2>Searching for meaning among the data</h2>
<p>Due to the choice to separate these out as separate objects in Elasticsearch,
most statistical information is straightforward to ascertain. Want to know
what formats are most popular? Look for <code>importVector</code> or <code>importRaster</code>
events and tabulate the number of times each format and/or driver are used.
Interested in how frequently a particular analysis tool is used? Look for
its corresponding event.</p>
<h2>When (and how) to pivot your data</h2>
<p>The one piece that had me stumped for a few days was: how do I determine how
many active users are on which version of macOS? I've got launch data and a
unique (but pseudonymous) <a href="https://www.macgis.com/privacy#hostid">host identifier</a>.
For example, you can create buckets based on the OS and count unique host IDs...
unfortunately, that creates a data problem with users who have upgraded during
the time period being examined. Using this technique, a user who was running
macOS 11 and upgrading on each release day would account for 10 separate macOS
counters.</p>
<p>What I really needed was to look at the data for just the most recent report
for each host ID. Basically, I needed to pivot around the host ID. After much
too much time trying to find a complex way through this, I finally searched on
"pivot elasticsearch" and found <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/transform-overview.html">Pivot Transformations</a>,
which turns out to be just what I needed. By creating a transformed index with
the latest method, I was able to get an index that only pointed to the most
recent documents for each host ID. Once I had this, I could aggregate using
<code>terms</code> to find the operating system, resulting in a bucket of OS versions used
most recently by each host ID.</p>
<p>Pivot Transformations create an alternate index to documents in another index.
In my case, I used a <code>latest</code> transformation, which maintains only the most recent
item for each unique key, based on the specified timestamp field and possibly
limited by a filter. In my case:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"ct-app-logs-*"</span>
<span class="w"> </span><span class="p">]</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"latest"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"unique_key"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"report.host.id"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"sort"</span><span class="p">:</span><span class="w"> </span><span class="s2">"report.timestamp"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Application Hosts"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"frequency"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1m"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"dest"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ct-app-hosts"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>creates the new index based on the existing <code>ct-app-logs-*</code> pattern and
pulling out items by the unique <code>report.host.id</code> key, using <code>report.timestamp</code>
to determine which item is the most recent. This boils down 12 indexes containing
18.5M documents into a single index containing 16K documents.</p>
<p>The destination index, <code>ct-app-hosts</code> was set up ahead of time using a basic
clone of the original index.</p>
<p>If desired, I could add a <code>query</code> key which would have limited the scope of the
documents.</p>
<h2>Upping the ante for aggregation</h2>
<p>Once I got this going, I was having some issues pulling information out of the
data due to some variances in how versions were managed. In particular, I was
interested in seeing major OS versions (macOS 10.15, macOS 11, iOS 14, etc.) and
maybe the same for the application version.</p>
<p>To facilitate this, I used runtime fields, setting up 2 additional mappings in
the destination index (<code>ct-app-hosts</code>) pointed at above. To do this, I <code>PUT</code> a
new index definition for the index, citing the following:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"mappings"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"runtime"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"major_app"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"type"</span><span class="p">:</span><span class="s2">"keyword"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"script"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"""</span>
<span class="s2">def myField = doc['report.application.version'];</span>
<span class="s2">if (myField.empty)</span>
<span class="s2"> emit("");</span>
<span class="s2">else {</span>
<span class="s2"> def dom = myField.value;</span>
<span class="s2"> for( String suffix : ['a','d','b']) {</span>
<span class="s2"> if (dom.indexOf(suffix)>0) {</span>
<span class="s2"> dom = dom.substring(0,dom.indexOf(suffix));</span>
<span class="s2"> }</span>
<span class="s2"> }</span>
<span class="s2"> int last = dom.lastIndexOf('.');</span>
<span class="s2"> if (last == dom.indexOf('.'))</span>
<span class="s2"> emit(dom);</span>
<span class="s2"> else</span>
<span class="s2"> emit(dom.substring(0,last));</span>
<span class="s2">}"""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"lang"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"painless"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"major_os"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"keyword"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"script"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"source"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"""</span>
<span class="s2">def myField = doc['report.host.os.version'];</span>
<span class="s2">if (myField.empty)</span>
<span class="s2"> emit("");</span>
<span class="s2">else {</span>
<span class="s2"> def dom = myField.value;</span>
<span class="s2"> int last = dom.lastIndexOf('.');</span>
<span class="s2"> def major = dom.substring(0,last);</span>
<span class="s2"> if (major.startsWith('11') || major=='10.16') {</span>
<span class="s2"> emit('11');</span>
<span class="s2"> } else {</span>
<span class="s2"> emit(major);</span>
<span class="s2"> }</span>
<span class="s2">}"""</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"lang"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"painless"</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
</code></pre></div>
<p>This creates a new column, <code>major_app</code> which:</p>
<ol>
<li>Checks that <code>report.application.version</code> exists</li>
<li>Removes any suffix starting with <code>a</code>, <code>b</code> or <code>d</code></li>
<li>Removes the last <code>.</code> (unless it is also the first <code>.</code>, such as in <code>1.4</code>)</li>
</ol>
<p>Similarly, it creates a column <code>major_os</code> which:</p>
<ol>
<li>Checks that <code>report.host.os.version</code> exists</li>
<li>Removes the last <code>.</code></li>
<li>Makes sure to emit <code>11</code> for <code>10.16</code></li>
</ol>
<p>With these two powerful tools, I was able to create a clear, concise, and
constantly-up-to-date resource for OS and application usage.</p>
Always check your arguments2021-04-24T13:16:00-04:002021-04-24T13:16:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-04-24:/always-check-your-arguments.html<p>Quite a while back, RS wrote a comprehensive ansible role for handling
<a href="https://letsencrypt.org">Let's Encrypt</a> certificate issuance and renewal.
We both use this role extensively, which is why it was a significant issue when it
suddenly started throwing type errors deep inside of the
<a href="https://www.dnspython.org">dnspython</a> library during an <code>nsupdate</code> call in …</p><p>Quite a while back, RS wrote a comprehensive ansible role for handling
<a href="https://letsencrypt.org">Let's Encrypt</a> certificate issuance and renewal.
We both use this role extensively, which is why it was a significant issue when it
suddenly started throwing type errors deep inside of the
<a href="https://www.dnspython.org">dnspython</a> library during an <code>nsupdate</code> call in a critical
part of the script.</p>
<p>A cursory examination of the component parts indicated that the most likely cause was
a change to the <code>dnspython</code> library, which had recently been upgrade from 1.16 to 2.0.
Although there wasn't anything we could find online indicating other people had suffered
this breakage (which should have been a clue), it hadn't been out very long, it crashed
in a module that indicated it was checking something with IPv6, we use a lot of IPv6 on
our systems, many people use no IPv6, and well, we hadn't changed anything...</p>
<p>This was an annoyance, but relatively easy to avoid in one of the following ways:</p>
<ul>
<li>Pin the <code>dnspython</code> libraries to <2.0 in <code>pip</code></li>
<li>On the Mac, use <code>brew</code>'s <code>ansible</code> and manually roll back the <code>dnspython</code> libraries
in the installed version<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup></li>
</ul>
<p>I used both, as we ran ansible on both SmartOS and macOS.</p>
<h2>Taking brew hackery up a notch</h2>
<p>After maintaining this for a while, I needed to upgrade some modules in <code>ansible</code>, and
needed to keep my CI environment (running on Macs under <a href="https://www.jenkins.io">Jenkins</a>
in sync with what we were running on my desktop, laptop, and servers; and that lead me to
<a href="https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap">create my own tap</a> in homebrew
by cloning the standard ansible formula and using my own repository.</p>
<p>The addition of this tap meant that I could configure this and test it once, but I could
deploy it on all of my Macs (and anyone else who had access to the tap on my private
git server could do the same).</p>
<h2>One thing leads to another</h2>
<p>After a few more months of using this tap on my Macs (and slowly moving ahead the <code>ansible</code>
version on the SmartOS machines, but keeping <code>dnspython</code> pinned), I needed to upgrade the
version of ansible at home (due to a project that I'll likely write about later, using
<code>ansible</code> to configure my Jenkins agents). The driver here was the need to execute
<code>homebrew</code> commands on an M1 mac, something that didn't work out of the box with <code>ansible</code>
2.9, which is what I was pinned to.</p>
<p>Ever-hopeful, I first decided to see if my aforementioned problem was "fixed" by unlinking<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>
my private tap's version of <code>ansible</code>, and installing homebrew's version.</p>
<p>Sadly, running the <code>ansible</code> playbook just resulted in the familiar crash. I looked at it
for a few minutes, decided the bug that was introduced in summer 2020 was still there and
set about building a new tap for version 3.2.0 of <code>ansible</code>. This went smoothly, but after
updating my formula, installing took a <em>long</em> time, on the order of a few minutes. Why was
the standard homebrew install so much faster?</p>
<h2>A bottle for monsieur?</h2>
<p>Quick investigation lead to the fact that most brew taps are installed these days using
bottles, or pre-built versions of the entire subdirectory that ends up in the Cellar. That
seemed like it was a significant win, especially since I was going to install this at least
5 times each update, so I decided to figure out how to create my own custom bottles for
my custom tap.</p>
<p>Thanks to a good article on <a href="https://web.archive.org/web/20201130235507/https://yehowshuaimmanuel.com/software_misc/custom_homebrew/cutsom_homebrew//">Custom Tap and Bottles with Homebrew</a>
by <a href="https://web.archive.org/web/20201130235507/https://yehowshuaimmanuel.com/">Yehowshua Immanuel</a>, I was on my way quickly after
rebuilding from my tap formula once for each platform of Mac that I run (Intel Catalina,
Intel Big Sur, and ARM Big Sur at this time).</p>
<h2>The final verdict</h2>
<p>After all this work, and getting a great solution in place for working around the
perceived bug in <code>dnspython</code>, I took another quick look at the bug that was popping up in
our role. I'd contributed to random python projects in the past and also contributed to
<code>ansible</code> directly, so I was familiar with the process and figured I could track the
problem down. I fired up <a href="https://www.jetbrains.com/pycharm/">pycharm</a> to get a little
better perspective on the particular bugs and settled in to reproduce a minimal set of the
problem with the <code>nsupdate</code> command in <code>ansible</code>.</p>
<p>A few minutes (literally) into the investigation and I found myself looking at the what
seemed like completely reasonable arguments to the <code>dns.query.tcp</code> method which were
raising exceptions due to not being able to determine whether my hostname was an IPv4
or IPv6 address. I immediately checked the current docs for <code>nsupdate</code> in <code>ansible</code> and,
indeed, the <code>server</code> argument is now designated an IP address (v4 or v6). Checking whether
we'd just been lucky and ignoring this all along, I went back to the <code>ansible 2.9</code>
documentation and verified that it was mute on the issue of what was in the string argument.</p>
<p>At some point between 2.9 of <code>ansible</code> and 3.0, they documented the change caused by
the the underlying library and I missed that change.</p>
<p>A few take-aways:</p>
<ol>
<li>Once again, a reminder that checking your arguments against current documentation is
often time well spent.</li>
<li>Assuming a behavior that goes against your expectations is a bug when nobody else
is complaining about it is often a recipe for a lot of work.</li>
<li>Homebrew is a really well thought out package and if you have a need to maintain your
own tools, it may be well worth it to use private taps and bottles, they're easy to
create and super-easy to use.</li>
</ol>
<p>Every once in a while, it's good to have your own assumptions challenged. I made a point
of commenting on the <a href="https://github.com/ansible-collections/community.general/issues/698">bug report</a>
for <code>ansible</code> regarding this filed by someone else. Hopefully they're find my information
useful.</p>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>This experience lead me to a nifty thing about <code>brew</code>, which is that many installations
have every dependency installed in the Cellar directory for that specific package, including (for most python tools),
it's own copy of site-packages. This makes it very easy to pin specific versions of dependencies
and be able to run a number of python tools with different libraries and even interpreters. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
<li id="fn2" class="footnote-item"><p>Everyone who uses <code>ansible</code> should be familiar with the <code>link</code> and <code>unlink</code> commands,
which allow you to keep a version or command installed while switching to another one. In
my case, since I was using a tap that had named versions (the best example of this I can
think of is Postgresql, which has separate versions for current, 12, 11, 10, 9.6 and even
some of the deprecated versions--use at your own peril). So, I could <code>brew unlink ansible@2.9.13</code>
and <code>brew install ansible</code> and get my private copy to move out of the way and use the
brew-standard version for testing. <a href="#fnref2" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
2021 Backup Software2021-03-28T10:42:00-04:002021-03-28T10:42:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-03-28:/2021-backup-software.html<p>As we approach another <a href="http://www.worldbackupday.com">World Backup Day</a>, I figured it
was time for me to revise my 2013 <a href="https://www.gaige.net/backup-software.html">backup article</a>
for a more up-to-date view of what my backup situation is and what I am currently
recommending.</p>
<p>My basic backup strategy, as outlined in the previous article, hasn't changed
significantly …</p><p>As we approach another <a href="http://www.worldbackupday.com">World Backup Day</a>, I figured it
was time for me to revise my 2013 <a href="https://www.gaige.net/backup-software.html">backup article</a>
for a more up-to-date view of what my backup situation is and what I am currently
recommending.</p>
<p>My basic backup strategy, as outlined in the previous article, hasn't changed
significantly; however the tools, services, and locations have. There's now general
acceptance now of the 3-2-1 backup strategy, which is:</p>
<ul>
<li>3 copies of your data (1 live, 2 backups)</li>
<li>2 different backup media/packages</li>
<li>1 copy offsite</li>
</ul>
<p>For the most part, I think that provides a good minimum basis; although I have been
adhering to that only as a minimum, and in particular I've been recommending an extra,
physically distant offsite location.</p>
<p>This year's post is big enough, it needs a TOC:</p>
<ul>
<li><a href="#Locations">Locations</a></li>
<li><a href="#Software">Software and Services</a></li>
<li><a href="#Encryption">Encryption</a></li>
<li><a href="#Photos">Photos</a></li>
<li><a href="#Conclusion">In Conclusion</a></li>
<li><a href="#OldSoftware">Afterward: Previous software suggestions</a></li>
</ul>
<h2>Locations <a id="Locations"></a></h2>
<p>With some reasonable bandwidth now widely available for most people who are using
computers, an offsite backup that is up-to-date should be considered table stakes.</p>
<p>Reiterating my advice from 7 years ago: a minimum of two physically distinct locations
is a must: one at home/on-site for ease of access, and at least one kept remote.
My last article suggested a safe deposit box, but I'm going to recommend against that
unless you don't have sufficient online access to systems physically diverse from
your location. Frankly, there's just too high of a likelihood that you won't remember
to update it. I suggest that the second copy be an online service, so that
you can make sure it is always available for you.</p>
<p>Again, I would suggest at least 2 locations. They should be far enough apart that
they're unlikely to suffer the same fate in the event of a disaster. If you are using
an online service for the second backup, try to find one that keeps two distinct copies
of your data (I'll describe one way to do that below).</p>
<h2>Software and Services <a id="Software"></a></h2>
<p>Last time, I described myself of being a big fan of paid-for backup software. Although
I'd agree with that in general still, I'll note that of the 5 packages that I called
out last time, there have been some changes.</p>
<p>Here are some specific suggestions:</p>
<h3>Time Machine <a id="TimeMachine"></a></h3>
<p>Time Machine is Apple's built-in backup software for many
versions of macOS, and is the only program that stayed on this list since the
last time. It provides version storage as well is very simple
administration, and can be used easily with an externally connected hard drive.
Of course, it's not very useful for off-site backup. However, for local backup
it is easy to set up and easy to restore data from.</p>
<h3>Arq</h3>
<p><a href="https://www.arqbackup.com">Arq</a> backup is a server-agnostic client-based offsite
backup package. They provide a software package that can use many different systems
for storage, including BackBlaze's B2, Amazon's S3 and Glacier, Dropbox, Google
Cloud and Drive, Microsoft OneDrive or SharePoint, external and network drives,
and just about any S3-compatible storage service (think <a href="https://min.io/">minio</a>,
<a href="https://wasabi.com">Wasabi</a> and others).</p>
<p>Personally, we're using Arq to back up to a pair of minio servers that we run:
one on each coast. We encrypt with complex keys at the client before the backups
are sent to the cloud, so we're confident that we're safe from prying eyes.</p>
<p>Arq's been around since 2009 and has been providing similar capabilities that whole
time. In the case of Arq, you are purchasing a software package (with updates if
you in maintenance) and you will need to provide separately for your storage.
Some might find that fiddly and a disadvantage.</p>
<p>For those wondering, there were some hiccups at the start of Arq v6 (first new release
since v5) and I felt that the author responded well to them. Lots of criticism,
as is the way of it in this day and age. His response was to buckle down and
accelerate the release of v7, which came out earlier this year, with a tuned-up
new Mac-native interface (one of the issues with v6 was the Electron interface).
I had no trouble with either v6 or v7, but I'm happy to be on v7 now and have
had good luck with backup and restore.</p>
<h3>Carbon Copy Cloner</h3>
<p><a href="https://bombich.com">Carbon Copy Cloner</a> is a package that clones Mac hard drives
(and SSDs, really any storage). They have been around a long time (since 2002) and shown
steady progress of solid software improvement. Even with the challenging
changes over the last few years for the Mac, the folks at Bombich Software have managed to
engineer their way around the new Apple choices with aplomb and have won their way back
to being my preferred disk cloning software. When I need to make a bootable (or
just carry-able) copy of an existing drive, I turn to Carbon Copy Cloner.</p>
<h3>Bacula <a id="Bacula"></a></h3>
<p>I'm a big proponent of Bacula for server backups. It requires a bit of an investment
up-front to figure out your backup plans and you need to be willing to put in the
time to understand the options and configuraiton. But, if you're looking for
something for servers, I would suggest checking my articles on Bacula:</p>
<ul>
<li><a href="https://www.gaige.net/welcome-bacula.html">Welcome Bacula</a></li>
<li><a href="https://www.gaige.net/bacula-6-months-on.html">Bacula 6 months on</a></li>
<li><a href="https://www.gaige.net/bacula-restore-testing.html">Bacula restore testing</a></li>
</ul>
<h3>BackBlaze</h3>
<p><a href="https://www.backblaze.com">BackBlaze</a> came onto the scene a number of years back and was
initially an also-ran behind CrashPlan and Carbonite at the time. My, how things
have changed. They're not perfect (note some recent Facebook-related unforced
errors on their web site), but they reportedly provide a reliable service and
charge reasonable prices. I've never been a customer of theirs for backup, but I
have used their S3-compatible storage system (B2) for offsite storage and found
them to be reasonable. They have the option of letting you self-key, which means
they won't be able to tell what you are backing up.</p>
<h2>Encryption <a id="Encryption"></a></h2>
<p>This is not an option. You need encryption. If you have enough operational
fortitude to keep your own keys, you should do that (as opposed to having
them escrowed by a backup provider).</p>
<p>It's especially important when keeping data off-site
to make sure that data is encrypted using strong encryption and with keys that
are only available to you. This is possible with some services like CrashPlan
and BackBlaze, and with the new entrant above, Arq.
Any data which is intentionally taken off-site should be stored in some
encrypted form. Keep in mind that if you designate your own keys, you are going
to have to safely store these keys in a
manner that they will not be lost by whatever event causes your data to be
lost. My suggestion is <strong>store a copy of your keys in a safe deposit box</strong>.
There are electronic methods of storing this, but why mess around? If your biggest
concern is losing the data, then keep those keys <em>unencrypted</em> in the safe deposit
box. If your biggest concern is somebody getting your data, then keep the keys
<em>encrypted</em> in the safe deposit box. With that said, in the case of a real disaster
(one you don't survive), determine how you want those keys to convey to your
heirs and assigns.</p>
<h2>Photos <a id="Photos"></a></h2>
<p>This year I wanted to add a separate item for photos (and video, for that matter).
Most people have a large amount of their data wrapped up in photos and videos
these days. You should treat this data basically as you should all valuable data
and have backups, in multiple locations, and with multiple methods.</p>
<p>Many of you may be using Google Photos or Apple Photos to store and manage your
photos. That's fair enough, but those services do little to prevent from
accidental (or malicious) destruction of photo data.</p>
<h3>Apple Photos</h3>
<p>If you're using Apple Photos to manage photos, you should consider having a single
machine with sufficient disk space to be designated for full-resolution backups. The
process is pretty simple:</p>
<ol>
<li>Log in to your AppleID on that machine (in your own account if it's a shared computer)</li>
<li>Start Photos</li>
<li>Choose <strong>Photos > Preferences</strong> and select the <strong>iCloud</strong> tab.</li>
<li>Under here, make sure the <strong>Download Originals to this Mac</strong> is checked</li>
<li>Back up your user account (or the location of the Photos library if you've moded it)
as you would any other valuable data</li>
</ol>
<h3>DSLR users</h3>
<p>If you're using a DSLR and RAW or very high resolution photography, you should
consider backing up that data one more time. I usually have a staging area for
photos while I'm on a trip (when we could take trips) and that tends to stay
around quite a bit longer than it theoretically needs to. It's a second set of
suspenders beyond the belt and suspenders that I'm already wearing.</p>
<h2>In Conclusion <a id="Conclusion"></a></h2>
<p>It doesn't really matter as much how you decide to back up your data, it just
matters that you do back up your data. If there's something that you care
about, back it up. If you care about the data being secure, encrypted it. If
for some reason you believe you care about the data and you don't care about
being secure, think again.</p>
<h2>Afterward: Previous software suggestions <a id="OldSoftware"></a></h2>
<p>I figured some of you may be interested in knowing the fate of recommended products
gone by. Since you may have followed my advice and chosen one or more of these, I'll
sum up the current thinking on each.</p>
<h3>CrashPlan</h3>
<p>I'd been a strong proponent of <a href="https://www.crashplan.com/en-us/">CrashPlan</a>
in the last article, and in some ways, I still like it.
The client, although "native", is still poorly
designed, but data integrity and speed are still fine.
What's changed is the business model. When I wrote this, you could still get a
personal subscription; now there are only "small business" and "Enterprise"
subscriptions, and outside of the "Enterprise" version, you're being pushed
toward using their backup service. That's fine, insofar as it goes, but at this
point, I think there are better players, like BackBlaze (above) for personal backup
at this point.</p>
<h3>SuperDuper</h3>
<p><a href="https://www.shirt-pocket.com/SuperDuper/SuperDuperDescription.html">SuperDuper!</a>
is a package that clones hard drives on the
Mac from one device to another. Over the years I have
switched back and forth between Carbon Copy Cloner and SuperDuper! Recently, the
folks at Shirt Pocket software have been slower to adapt to Apple's changes and
I'm currently in a Carbon Copy Cloner phase, and that's what I recommend.</p>
<h3>BRU Server</h3>
<p>The company announced they were <a href="https://www.gaige.net/images/Tolisgroup-SOL.png">going out of business</a>;
the product was later purchased by <a href="https://macsales.com">OWC</a>.
Maybe OWC will do something with it in the years to
come, but I no longer advise its use. If you're looking for something for servers,
I would suggest checking the section above on <a href="#Bacula">Bacula</a>.</p>
<h3>Retrospect</h3>
<p>Historically (in the old days), I used <a href="https://www.retrospect.com">Retrospect</a>,
which went down hill significantly when the Dantz was acquired by EMC.
The software product was spun back out into Retrospect, Inc. in November of
2011, and the word is that it has improved markedly since then.
I gave it a try again once, but have not used it in production,
nor have I tried recent versions.</p>
pre-commit and Pelican2021-03-28T09:00:00-04:002021-03-28T09:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-03-28:/pre-commit-and-pelican.html<h2>Putting pre-commit to use</h2>
<p>I mentioned in a <a href="https://www.gaige.net/pre-commit.html">previous post</a> about pre-commit, a
tool for maintaining code consistency through simple management of pre-commit checks.</p>
<p>The first place I decided to give this a whirl was on my blog sites. As you may be aware,
I moved my blog sites (both …</p><h2>Putting pre-commit to use</h2>
<p>I mentioned in a <a href="https://www.gaige.net/pre-commit.html">previous post</a> about pre-commit, a
tool for maintaining code consistency through simple management of pre-commit checks.</p>
<p>The first place I decided to give this a whirl was on my blog sites. As you may be aware,
I moved my blog sites (both <a href="https://gaige.net">Gaige's Pages</a> and <a href="https://blog.cartographica.com">The Cartographica Blog</a>)
to <a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">static sites</a> some time back.</p>
<p>Pelican markdown files have a preamble that is set apart by a blank line. Basically,
a set of colon-delimited key-value pairs that are rudimentarily parsed and then passed
to the interpreter. Basically it looks like this:</p>
<div class="codehilite"><pre><span></span><code><span class="nl">Title</span><span class="p">:</span><span class="w"> </span><span class="n">My</span><span class="w"> </span><span class="n">Blog</span><span class="w"> </span><span class="n">Post</span>
<span class="nl">Date</span><span class="p">:</span><span class="w"> </span><span class="mi">2021</span><span class="mo">-03</span><span class="mi">-28</span><span class="w"> </span><span class="mo">07</span><span class="o">:</span><span class="mi">48</span>
<span class="cp"># Some bloggy stuff</span>
<span class="n">Content</span><span class="w"> </span><span class="n">text</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">here</span><span class="p">...</span><span class="w"> </span><span class="n">Oh</span><span class="p">,</span><span class="w"> </span><span class="n">see</span><span class="w"> </span><span class="n">my</span><span class="w"> </span><span class="p">[</span><span class="n">previous</span><span class="w"> </span><span class="n">post</span><span class="p">]({</span><span class="n">filename</span><span class="p">}</span><span class="n">previous</span><span class="o">-</span><span class="n">post</span><span class="p">.</span><span class="n">md</span><span class="p">)</span>
</code></pre></div>
<p>In addition to the formatter, there are also some replacement items that can be used to
reference generated data. For example: <code>{filename}</code> indicates that the path to the
stored file should be substituted.</p>
<p>I had noticed there was a Markdown plugin for pre-commit using
<a href="https://github.com/executablebooks/mdformat">mdformat</a>, and so I figured I'd give that a
try. Initial results were good. It provided a lot of clean-up for free. On the downside:
it also quoted all of the <code>{filename}</code> and similar references, such that they would no
longer work as references. And, it also eliminated my footnotes.</p>
<p>My initial <code>.pre-commit-config.yaml</code> looked like this:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># See https://pre-commit.com for more information</span>
<span class="c1"># See https://pre-commit.com/hooks.html for more hooks</span>
<span class="nt">repos</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/pre-commit/pre-commit-hooks</span>
<span class="w"> </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v3.2.0</span>
<span class="w"> </span><span class="nt">hooks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">trailing-whitespace</span>
<span class="w"> </span><span class="nt">exclude</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">^.*\.md$</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">end-of-file-fixer</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-yaml</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-added-large-files</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-json</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/executablebooks/mdformat</span>
<span class="w"> </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.5.7</span>
<span class="w"> </span><span class="nt">hooks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat</span>
<span class="w"> </span><span class="c1"># optional</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'--number'</span>
<span class="w"> </span><span class="nt">additional_dependencies</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat-tables</span>
</code></pre></div>
<p>Note here a couple of items:</p>
<ul>
<li>I have excluded <code>^.*\.md$</code> from <code>trailing-whitespace</code>, this was specifically to deal
with the fact that I had some two-space-at-end-of-previous-line implementations for handling
forced line-breaks. This is one of a few ways of doing this, but was required for use with
the <code>python-markdown</code> module that's used by default with pelican</li>
<li>I have added the <code>mdformat</code> plugin with a number of options and dependencies</li>
<li><code>--number</code> as an argument to <code>mdformat</code> forces it to number ordered list items. I prefer
that for readibility.</li>
<li><code>mdformat-tables</code> adds table handling to mdformat (by default it uses a strict version
of Markdown called <a href="https://commonmark.org">Commonmark</a>), so any extensions must be enabled
with intention</li>
</ul>
<h2>mdformat plugins</h2>
<p>With things mostly working, I looked at the <code>mdformat</code> documentation to see if I could
make changes to the way it operated. Fortunately, there was a plug-in architecture that
allowed for the modification of both parsing and output behavior.</p>
<h3>Footnotes</h3>
<p>Although there's support for footnotes in the underlying markdown parser that's used
by <code>mdformat</code>, that parser (<a href="https://github.com/executablebooks/markdown-it-py">markdown-it-py</a>,
based on the Javascript-based <a href="https://github.com/markdown-it/markdown-it">markdown-it</a>),
that support wasn't built-in to the mdformat code. So, I decided that I'd take a look
at <code>mdformat-tables</code> and see if I could do something similar for footnotes, since
the code for both tables and footnotes are included in the underlying package as options.</p>
<p>The result is the <a href="https://github.com/gaige/mdformat-footnote">mdformat_footnote</a> plugin,
which uses the existing parser (the hard part) and formats the footnotes appropriately.</p>
<p>This plugin can be installed using <code>pip install mdformat_footnote</code> or by adding <code>mdformat_footnote</code>
to the list of items in the <code>additional_dependencies</code> list in the <code>.pre-commit-config.yaml</code>
file.</p>
<h3>Pelican-specific items</h3>
<p>In this initial case,
all I needed to do was change the output so that it didn't replace the <code>{}</code> characters
inside of links. The code was straightforward, and after some playing around, I created
the <a href="https://github.com/gaige/mdformat-pelican">mdformat_pelican</a> plugin for use with
mdformat and pelican.</p>
<p>You can look at the code above, or install it with <code>pip mdformat_pelican</code> to get
the latest version from <a href="https://pypi.org">pypi.org</a>.</p>
<p>Implementing the <a href="https://github.com/gaige/mdformat-pelican/tree/v0.0.2">initial code</a> was
straightforward. Effectively, the code hijacks the <code>render_token</code> function and modifies
the <code>token.attrs</code> just before they're rendered, correcting any erroneously-quoted
URLs.</p>
<p>This worked great, across nearly all of my files. Except for a couple that had square
brackets in their metadata fields. For example, a post about Queen guitarist Brian
May receiving his doctorate in Astrophysics had this front matter:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">Date</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2007-08-03 07:26</span>
<span class="nt">Alias</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/node/4836,/article.php?story=20070803092627654</span>
<span class="nt">Tags</span><span class="p">:</span>
<span class="nt">Category</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">general news</span>
<span class="nt">Title</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="nv">He's</span><span class="p p-Indicator">]</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">a killer... astrophysicist?</span>
</code></pre></div>
<p>which <code>mdformat</code> dutifully turned into <code>\[He's\] a killer... astrophysicist?</code>,
which <code>pelican</code> didn't know how to interpret, so the backslashes ended up in my content
pages...not desired.</p>
<p>Since I already had a Pelican plugin for <code>mdformat</code>, I decided to make it a bit more
pelican-y, by marking the front matter as off-limits. This was a little trickier, but
had good results. As you can see in the <a href="https://github.com/gaige/mdformat-pelican/blob/master/mdformat_pelican/plugin.py">plugin source</a>,
understanding the frontmatter required adding a parser by putting in a new block rule and
then putting in the parser as well as the code to render that later in <code>render_token</code>.</p>
<p>Since the format is very rigid (basically, collect everything until you reach the first
blank line), it was easy to implement.</p>
<p>So, now my current working <code>.pre-commit-config.yaml</code> looked like this:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># See https://pre-commit.com for more information</span>
<span class="c1"># See https://pre-commit.com/hooks.html for more hooks</span>
<span class="nt">repos</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/pre-commit/pre-commit-hooks</span>
<span class="w"> </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v3.2.0</span>
<span class="w"> </span><span class="nt">hooks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">trailing-whitespace</span>
<span class="w"> </span><span class="nt">exclude</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">^.*\.md$</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">end-of-file-fixer</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-yaml</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-added-large-files</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-json</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/executablebooks/mdformat</span>
<span class="w"> </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.5.7</span>
<span class="w"> </span><span class="nt">hooks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat</span>
<span class="w"> </span><span class="c1"># optional</span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'--number'</span>
<span class="w"> </span><span class="nt">additional_dependencies</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat-tables</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat-black</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat_footnote</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mdformat_pelican</span>
<span class="nt">exclude</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">|</span>
<span class="w"> </span><span class="no">(?x)(</span>
<span class="w"> </span><span class="no">^output/|</span>
<span class="w"> </span><span class="no">^themes/|</span>
<span class="w"> </span><span class="no">^venv/|</span>
<span class="w"> </span><span class="no">^content/NewZealand/</span>
<span class="w"> </span><span class="no">)</span>
</code></pre></div>
<p>This adds my new plugins (both the <code>mdformat_footnote</code> and the <code>mdformat_pelican</code>) and
also adds an exclusion for some files in my pre-commit hooks. The ones that aren't actually
committed (<code>output</code>, <code>venv</code>) wouldn't be included, but I have a set of badly-formatted
HTML files in <code>content/NewZealand</code> that I don't want to fix yet.</p>
<p>This turned out well, but I had a couple of items that the parser in Pelican and the
parser in mdformat could not agree on. In particular, things like indentation
requirements for items with newlines within ordered lists that have multiple paragraphs
in them.</p>
<p>In the end, that would lead me to write a new plugin for Pelican to replace the
Markdown parser.</p>
<h2>Markdown parser plugin for Pelican</h2>
<p>The plugin architecture for <code>mdformat</code> is pretty good, but the one for Pelican is very
mature and well thought-out. I've created plugins for Pelican before, notably the
<a href="https://www.gaige.net/pelican-plugin-for-nginx-redirection.html">Nginx alias maps plugin</a>.</p>
<p>Also, there already existed plugins to replace the Markdown reader in Pelican. As such
the lift was pretty light:</p>
<ol>
<li>Get a base plugin working</li>
<li>Parse the metadata (simple <code>:</code> split of each line before the first blank line)</li>
<li>Load the <code>MarkdownIt</code> package and configure with a few settings (tables, footnotes, and definition lists)</li>
<li>Add hooks to rewrite the <code>\{filename\}</code> items back to <code>{filename}</code></li>
<li>Finally, add a new <code>fence</code> formatter, to use <a href="https://pygments.org">Pygments</a> to format code</li>
</ol>
<p>The code is available on <a href="https://github.com">GitHub</a> in the <a href="https://github.com/gaige/markdown-it-reader">markdown-it-reader</a>
repository and can be installed using <code>pip install pelican-markdown-it-reader</code>.</p>
<p>This plugin must be enabled on your site by adding it to the list of <code>PLUGINS</code> in
your <code>pelican.py</code> file.</p>
pre-commit2021-03-28T08:00:00-04:002021-03-28T08:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2021-03-28:/pre-commit.html<h3>Introducing pre-commit hooks</h3>
<p>I recently became aware of the open-source project: <a href="https://pre-commit.com">pre-commit</a>,
which is "A framework for managing and maintaining multi-language pre-commit hooks."</p>
<p>The key feature of pre-commit is that it creates an execution environment for itself in
order to enable running hooks without messing with (or creating conflicts with …</p><h3>Introducing pre-commit hooks</h3>
<p>I recently became aware of the open-source project: <a href="https://pre-commit.com">pre-commit</a>,
which is "A framework for managing and maintaining multi-language pre-commit hooks."</p>
<p>The key feature of pre-commit is that it creates an execution environment for itself in
order to enable running hooks without messing with (or creating conflicts with) your
development or operating environment.</p>
<p>pre-commit uses a configuration file (<code>.pre-commit-config.yaml</code>) to determine which
hooks to run and how to handle them. These hooks can come from public or private
repositories and there's a pretty solid mechanism for dealing with dependencies.</p>
<p>I'd discovered the pre-commit project when looking at another open-source project that
used it to maintain a level of code consistency. To carry this out, pre-commit can be
used to update files during the commit phase to bring those files into alignment with
the standards (and verify correctness, run lint programs, etc.)</p>
<p>Pre-commit can be installed on macOS using <a href="https://brew.sh/">homebrew</a> with</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># brew install pre-commit</span>
</code></pre></div>
<p>Once installed, you can add a pre-commit configuration by creating a <code>.pre-commit-config.yaml</code>
file and putting appropriate contents in it. Here's an example config that cleans up yaml,
end of file, and trailing whitespace:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">repos</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">repo</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">https://github.com/pre-commit/pre-commit-hooks</span>
<span class="w"> </span><span class="nt">rev</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v2.3.0</span>
<span class="w"> </span><span class="nt">hooks</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">check-yaml</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">end-of-file-fixer</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">id</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">trailing-whitespace</span>
</code></pre></div>
<p>Install the git hook scripts with the install command:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># pre-commit install</span>
</code></pre></div>
<p>And now give it a shot by running the command against all of your files:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># pre-commit run -a</span>
</code></pre></div>
<p>It's not necessary to use <code>pre-commit run</code>, since the whole point of the pre-commit hooks
is that they run before code is committed. However, if you make a change to your pre-commit
configuration and want to bring the older content into line, you can use <code>pre-commit run</code> to
clean up existing code to the new standards.</p>
<p>All told, it's pretty spiffy. I decided to use it to
<a href="https://www.gaige.net/pre-commit-and-pelican.html">check my blog content</a>,
but that's a story for another <a href="https://www.gaige.net/pre-commit-and-pelican.html">post</a>.</p>
Bacula Restore Testing2020-09-27T10:33:00-04:002020-09-27T10:33:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-09-27:/bacula-restore-testing.html<p>Originally this was going to contain a brief <a href="https://www.gaige.net/bacula-6-months-on.html">Bacula, 6 months on</a>
section at the start. Of course, that became much too detailed, so I split them up, however
I would encourage you to read it.</p>
<h2>Restore Testing</h2>
<p>Backup is the most obvious part of doing backups. Almost everyone's aware …</p><p>Originally this was going to contain a brief <a href="https://www.gaige.net/bacula-6-months-on.html">Bacula, 6 months on</a>
section at the start. Of course, that became much too detailed, so I split them up, however
I would encourage you to read it.</p>
<h2>Restore Testing</h2>
<p>Backup is the most obvious part of doing backups. Almost everyone's aware they need to
perform them and so most businesses and some individuals make sure that they have one or
more backup systems and destinations.</p>
<p>However, it's surprising how infrequently real restore tests get performed. Frequently
this is because of the difficulty of shutting down a production system for verification
or because the commands are difficult, or even that it's difficult to find the available
disk space necessary to do a complete restore.</p>
<p>With that said, restore testing is absolutely crucial and becomes even more so as your
backup systems become more complex. However, it also gets quite a bit more difficult the
more complex your systems becomes, and as such hasn't always been something that I've done
well.</p>
<p>Fortunately, one of the side-effects of moving to Bacula was that we needed to perform
backup and restore tests when we were considering the solution and that has resulted in
some solid steps for performing restore tests.</p>
<h2>Verify vs Restore</h2>
<p>Frequently, you'll see "Verify" options along with backups and often even default read-after-write
operations which can validate that the data written is what was expected. Those options,
especially the latter, were absolutely essential in the days of tape backups, where you'd
occasionally have a tape go bad during the writing process. However, these days, the need
for verify-after-write is not nearly as strong.</p>
<p>Similarly, you'll see "Verify" operations which will attest to the fact that the data that
is stored on your backup "volumes" is what is currently on your machine. These are sometimes
used as a substitute for actual restore testing, as they simulate the restore process. In
effect, they exercise the catalog, validate the contents of the backup (sometimes) and
compare to the bytes in the file on disk (also sometimes). These are good to a point, but really
no substitute for doing what you're going to do in an emergency.</p>
<p>Performing actual restores on a regular basis not only exercises all mechanisms of the storage
and restore process, but they also tend to lead to automating your disaster recovery scenarios
and improving familiarity with the details of the restore process.</p>
<h2>Near In-place restore</h2>
<p>Because of the way that Bacula is structured, a client must exist in order to be the target
of a restore. In addition, it must contain the correct encryption keys. As such, the most
straightforward restore test is to restore the contents of your volumes to another location
on the existing client. As long as you have enough storage space, this is a pretty
non-invasive restore and can be done by using the standard restore commands and adding
the <code>where=</code> argument (or adjusting the restore parameter's <code>where</code> value). After running
the restore, use your favorite diffing utility to determine if everything is as it should
be and you've got a basic restore verification.</p>
<h2>Restoring to staging</h2>
<p>A more complex scenario involves restoring to a staging device. With our current configuration
of explicitly separate staging and production environments, this can be a little tricky, but
it has the advantage of allowing you to do an in-situ replacement and validate the full
restoration process.</p>
<p>Restoring to a "similar" production device would serve nearly the same purpose, but in our
case, the staging systems are designed to be safe copies of the production environment, and
there's very little "stream crossing" to be done.</p>
<p>With that said, if your staging and production environments are separate, you'll need to
do the following in order to restore on a staging system:</p>
<ol>
<li>Register your staging system with your production bacula director by adding a client
stanza for it in the <code>bacula-dir.conf</code> and <code>reload</code> the config in <code>bconsole</code></li>
<li>Place the encryption key (public and private halves) on your staging server, so the data
will decrypt appropriately</li>
<li>Add the production director to your <code>bacula-fd.conf</code> on the staging client, so that it
can request the restoration</li>
<li>Restart the File Daemon on the staging client</li>
<li>Use <code>status client</code> to make sure that you can reach the client</li>
<li>(Belt and suspenders) I like to also disable the File Daemon on the original client to
guarantee that you don't fat finger something and make yourself very unhappy</li>
<li>Run the restore command in <code>bconsole</code> selecting the appropriate files, setting the
client to your staging server, and setting the restoration path to <code>/</code> so that you
write to the original location</li>
<li>Once complete, remove the staging server from your production director's <code>bacula-dir.conf</code>
(to avoid confusion or accidental backups in the future), and <code>reload</code> in <code>bconsole</code>.
You may also want/need to delete the client in <code>bconsole</code>, which will make sure it's not
available.</li>
<li>Replace the configuration on the staging device as necessary</li>
</ol>
Bacula 6 months on2020-09-26T08:04:00-04:002020-09-26T08:04:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-09-26:/bacula-6-months-on.html<p>It's been about six months since I originally wrote <a href="https://www.gaige.net/welcome-bacula.html">Welcome Bacula</a>,
describing our transition to Bacula from our previous solution (and a bit of history even before that). If you
haven't read it, it might be worth a read.</p>
<p>Although not quite 6 months since I wrote the first piece …</p><p>It's been about six months since I originally wrote <a href="https://www.gaige.net/welcome-bacula.html">Welcome Bacula</a>,
describing our transition to Bacula from our previous solution (and a bit of history even before that). If you
haven't read it, it might be worth a read.</p>
<p>Although not quite 6 months since I wrote the first piece, it's now been over 6 months since
we started using <a href="https://bacula.org">Bacula</a>. The results have been extremely good:</p>
<ul>
<li>Performance has been excellent</li>
<li>The backup mechanism has been highly reliable</li>
<li>Locally-cached cloud backups propagate to the cloud easily (and reliably)</li>
<li>Pre-transmission compression and encryption have improved performance and security</li>
<li>Text-based configuration files have improved automation of clients and servers</li>
</ul>
<h3>Performance</h3>
<p>I'm going to start with performance, which has been an unexpected (and uncontested) win
in comparison to our previous solution. Nightly incremental backups are finishing in 10-12
minutes after backing up a bit over 3GB in 2500 files across 17 machines (remember that
a lot of systems we have don't store persistant data). Weekly differentials take a few minutes
longer and tend to contain about double the number of files and data. Full
backups take around 21 hours, backing up 350GB in 3 million files (including local storage
and the push to the offsite storage).</p>
<p>With our previous solution, we carved out a 10 hour window for full backups for
each of 4 backup sets covering 13 systems (the ease of automation in Bacula has resulted
in our backing up a few more machines), and about an hour a day for each of the 4 backup
sets. They didn't always take that long, but running the backup sets in parallel was
not a good idea™. Full backups took about 41 hours (total, not running in parallel) to
back up about 325GB. These backups were compressed, but not encrypted. In addition, the
push to offsite storage was a separate operation and itself took a substantial amount of
time (including requiring an encryption step). Generally, we could expect full backups to
be completed, encrypted, and replicated to offsite within 48-72 hours of the start of the
cycle.</p>
<h3>Reliability</h3>
<p>We've had excellent reliability out of Bacula. Error messages are delivered in a timely
fashion (via email mostly) and status information during the job is readily accessible.
The previous solution never had any reliability problems (that we were aware of), although
getting real-time status information was always a bit of a chore. The GUI (which Bacula
does not have) was a underwhelming and vague, but the CLI was too machine-friendly.
In this case, Bacula aligns much better with our needs and desires. I'm very comfortable
with CLIs and although GUIs are nice when they're done really well, for a facility like this
I'd rather have a good CLI any day.</p>
<h3>Cloud experience</h3>
<p>I need to give the caveat here that we don't use what most people would call "the cloud"
these days, as we have enough geographic diversity that we replicate to our own equipment
in another data center. However, the concepts are the same, and since we're using an
S3-compatible storage mechanism, I think the comparison to S3 or B2 is reasonable.</p>
<p>Bacula uses a fairly intelligent cloud cache which uploads backups in chunks as they are
completed. I'm still not entirely certain whether this stops the backup process in order
to upload or whether it uploads in parallel. Given that the backup isn't considered finished
until the cloud send has been attempted, it doesn't make much difference to us. You'll
note that I said "has been attempted". In the event that the cloud send fails, the backup
continues and an error is logged. You can attempt to upload the parts later if they aren't
completed before the backup.</p>
<p>It's worth noting that the cloud backups are just simple syncs of the directories from
the cache, so you can actually use any mechanism you like to send them off site. However, using
the built-in drivers also allows the system to pull the backups (piecemeal as required) during
a restore, which is a nice feature if you're bandwidth constrained either in network or
pricing.</p>
<p>I'll note here that the <code>part.1</code> of the backup (inside of the "volume" directory) is the
label and is required for the automatic pull to work. If that's deleted for some reason and
you need to restore from a volume that's completely offline, you'll need to at least pull
the <code>part.1</code> file back to the storage server's cache to get the automatic pull to work.</p>
<p><em>Ed Note: this previously read <code>part.0</code>, which isn't actually created by Bacula</em></p>
<h3>Automating configuration</h3>
<p>As should be clear from the rest of this blog, Rob and I use Ansible for building basically
everything we run. In many ways, the most significant advantage of the move to Bacula was
being able to automate the configuration of both clients and servers without difficulty.
As such, we have test and production environments and we're able to validate new versions
and configuration ideas when we need to.</p>
<h3>Conclusion</h3>
<p>All told, as with any good migration of a long-standing system, the main take-away is: I
wish I'd done this sooner. Bacula may be too fiddly for some people, but our environment
is complex and highly automated. As such, we constrained the fiddling mostly to our initial
configuration and have been able to craft a solution that is well suited to our environment
and needs.</p>
Trapped in the ice2020-06-26T06:29:00-04:002020-06-26T06:29:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-06-26:/trapped-in-the-ice.html<p>We've heard it all before: AWS is expensive, and you need to watch out for the hidden
sharp edges of their pricing model. Today I provide a small lesson in that concept.</p>
<h1>History</h1>
<p>ClueTrust has run through a number of backup methodologies over the year, originally
using Retrospect (when they …</p><p>We've heard it all before: AWS is expensive, and you need to watch out for the hidden
sharp edges of their pricing model. Today I provide a small lesson in that concept.</p>
<h1>History</h1>
<p>ClueTrust has run through a number of backup methodologies over the year, originally
using Retrospect (when they were their original, independent, selves) to tape, then moving to
BRU to handle more multi-platform capabilities, eventually deprecating tape and mirroring
to an off-site storage system, and most recently, our <a href="https://www.gaige.net/welcome-bacula.html">move to Bacula</a>.</p>
<p>BRU didn't have a glacier module, so I wrote (and re-wrote) a series of python scripts
that handled backing up, storing metadata (because glacier doesn't allow you to choose
the names of your storage units) and purging older archives when appropriate.</p>
<h1>Melting the glacier</h1>
<p>As part of our work this year, we've been looking at various storage models for our new
datastore. Since Bacula is capable of supporting S3, we looked at storing off-site data
using S3-compatible servers in a couple of locations. On the open-source side, this is
powered by <a href="https://min.io">minio</a>, but we also considered using the new <a href="https://backblaze.com">Backblaze</a>
<a href="https://www.backblaze.com/b2/docs/s3_compatible_api.html">S3 Compatible API</a>.</p>
<p>Either way, it was clear that raw glacier, as we'd been doing in the past, wasn't going
to make any sense for us going forward.</p>
<p>In the intervening years, Amazon had done a nice job of reducing the price of storage, and
even retrieval (in bulk) for Glacier, and we only have about 3TB of data sitting there right
now. This costs us approximately $12/month to store. Still $144/year, which at today's prices
will get you just about 1.5 4TB SMR drives per year (don't get me started on SMR, <a href="https://www.youtube.com/watch?v=8hdJTwaTl8I">especially
since we use ZFS</a>).</p>
<p>We're just beyond the 90-day window of deleting data from Glacier, so I took a look at what
it would cost us to download and archive the data from Glacier locally and just delete the
rest of it. For those of you unfamiliar with Glacier, there's a minimum 90-day retention
policy; if you delete your data in <90 days, you pay for the entire 90 days for that data.</p>
<h1>Getting you coming and going</h1>
<p>This section title is mildly misleading: AWS doesn't charge you for upload (except transactions),
but they do charge for storage ($0.004/GB/mo right now), and transfers out of Glacier to
the internet ($0.09/GB).<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup></p>
<p>So, to offload my 3TB of data it would cost 0.09 * 3000 or $270 (+$7.5 for the bulk retrieval fee).</p>
<p>We don't have all of that data locally stored (retention policies are tricky, and glacier
guarantees a certain level of redundancy), so we will slowly delete that data as it ages
out of our retention policy and hope that we don't need to restore it (and pay
the retrieval fee). So, glacier's got us as a customer for a few more months declining from
$12/month due to the financial lock-in of the retrieval price.</p>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>Pricing as of 2020-06-26 06:49. If you're reading this more than 6 months from now, pricing has probably gone down again. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
Static pages-18 months on2020-05-31T21:31:00-04:002020-05-31T21:31:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-05-31:/static-pages-18-months-on.html<p>In 2018, I wrote about the move to
<a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">convert Gaige's Pages to a static generation model</a>.</p>
<p>I <a href="https://www.gaige.net/follow-up-on-static-pages.html">followed this up</a> in mid-December of that year
describing the drop in processing and response time.</p>
<p>After 18 months of running the site (and Cartographica's Blog as well) on Pelican, I wanted
to …</p><p>In 2018, I wrote about the move to
<a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">convert Gaige's Pages to a static generation model</a>.</p>
<p>I <a href="https://www.gaige.net/follow-up-on-static-pages.html">followed this up</a> in mid-December of that year
describing the drop in processing and response time.</p>
<p>After 18 months of running the site (and Cartographica's Blog as well) on Pelican, I wanted
to revisit the graph and look at today's performance.</p>
<a href="https://www.gaige.net/images/large/2020-05-31-Performance-gaigespages.png">
<img src="https://www.gaige.net/images/large/2020-05-31-Performance-gaigespages.png" width=619 height=212>
</a>
<p>Not only are things still looking super zippy (average time to deliver of 127ms), but it's
pretty darned consistent. Not surprisingly, most of the time (90+% of it) is spent doing
the TLS handshake.</p>
<p>In December, 2018 (on the same hardware) we were seeing an average of 125ms
(minimum of 18ms, maximum of 805ms), now those numbers are almost unchanged with the average
of 127ms (minimum of 18ms, and a maximum of 178ms).</p>
<p>In the days of the wild west of the internet (pre-SSL/TLS), this site would have been
delivering in an average of 20ms.</p>
XCTest + CoreData = ouch2020-05-31T17:55:00-04:002020-05-31T17:55:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-05-31:/xctest-coredata-ouch.html<p>I put this up in hopes that somebody runs across it more quickly than I did...</p>
<p>This weekend, as a "break", I decided to do some work updating an ancient (2003-vintage)
piece of code that I wrote when I was doing extensive blogging. I'm not certain it'll ever
leave my …</p><p>I put this up in hopes that somebody runs across it more quickly than I did...</p>
<p>This weekend, as a "break", I decided to do some work updating an ancient (2003-vintage)
piece of code that I wrote when I was doing extensive blogging. I'm not certain it'll ever
leave my computers, but it was an opportunity to play around with some technologies that
I'd honestly not touched in years, including CoreData.</p>
<p>Among the things that I did was modify my code to use the more modern <code>NSPersistentContainer</code>,
in hopes that I could experiment some with CloudKit. Although it's likely that I'll do that
manually, at least at first, the thought of trying out the latest way of doing this made
sense (to me, at the time).</p>
<p>Unfortunately, I have a habit of writing unit tests. I say unfortunately not because I don't
see enormous value in them (in fact, I uncovered a long-standing bug in the existing code
with the first test that I wrote). I say unfortunately, because a lot of people don't write
them, or don't take them as seriously, and that means that strange interactions between
tests and other parts of the OS tend to be harder to find.</p>
<p>In this case, the <code>NSPersistentContainer</code> came back to bite me hard. As I later located,
there are some other people who have <a href="https://forums.raywenderlich.com/t/multiple-warnings-when-running-unit-tests-in-sample-app/74860/5">seen this problem</a>,
and it caused some <a href="https://stackoverflow.com/questions/51851485/multiple-nsentitydescriptions-claim-nsmanagedobject-subclass">problems for them</a>
as well.</p>
<p>In my case, I'd already created a container for my CoreData stack, and that was part of what
caused me the pain. In order to test that independently, I created a test bundle which executed
stand-alone. This bundle (of course) had a copy of the model file. Here's where my problems started.</p>
<p>Due to things going on behind the scenes and differences between Objective-C's and Swift's
handling of namespaces (oh, I didn't mention that I was also doing all the new code in
swift, with an existing, albeit small, Objective-C code base?) I had to make sure that
I was passing the <code>managedObjectModel</code> parameter to my <code>NSPersistentContainer</code> init method
so that it would find the right one (otherwise in the test bundle it would fail entirely).</p>
<p>In order to do this, I needed code to grab the bundle of the class I was using so that I
could guarantee that the right bundle was being loaded. Easy enough, I wrote a small class
method:</p>
<div class="codehilite"><pre><span></span><code><span class="kd">static</span> <span class="kd">func</span> <span class="nf">managedObjectModel</span><span class="p">()</span> <span class="p">-></span> <span class="bp">NSManagedObjectModel</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nv">bundle</span> <span class="p">=</span> <span class="n">Bundle</span><span class="p">(</span><span class="k">for</span><span class="p">:</span> <span class="kc">self</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">NSManagedObjectModel</span><span class="p">.</span><span class="n">mergedModel</span><span class="p">(</span><span class="n">from</span><span class="p">:</span> <span class="p">[</span><span class="n">bundle</span><span class="p">])</span><span class="o">!</span>
<span class="p">}</span>
</code></pre></div>
<p>and in my initializer for my container class, I loaded up the container:</p>
<div class="codehilite"><pre><span></span><code><span class="kd">let</span> <span class="nv">container</span> <span class="p">=</span> <span class="bp">NSPersistentContainer</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="s">"SiteQuoter"</span><span class="p">,</span> <span class="n">managedObjectModel</span><span class="p">:</span> <span class="n">SQExceptionManager</span><span class="p">.</span><span class="n">managedObjectModel</span><span class="p">())</span>
</code></pre></div>
<p>Which worked great. For my first test; and on every subsequent test started throwing non-fatal
warnings. Admittedly, I have a real problem with warning, both at compile time and run time
and I don't like them in tests either, so I spent too much time trying to track this down.
Note: the tests succeeded despite the warning that:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">Failed</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">find</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">unique</span><span class="w"> </span><span class="nv">match</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">an</span><span class="w"> </span><span class="nv">NSEntityDescription</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">managed</span><span class="w"> </span><span class="nv">object</span><span class="w"> </span><span class="nv">subclass</span>
</code></pre></div>
<p>After looking around (see above links), I confirmed that other people were also seeing
the unexpected caching of the <code>NSManagedObjectModel</code> and set about to make sure I only
created one of them (hoping that would solve my <code>NSPersistentContainer</code>-related problem).</p>
<p>I won't disclose exactly how long it took to figure out the magic incantation, but I will
disclose that Xcode's code analysis system was crashing most of the time that I wasn't doing
it right, and thus "functionality was limited".</p>
<p>In the end, I replaced my static method with an lazily-initialized class variable:</p>
<div class="codehilite"><pre><span></span><code><span class="kd">static</span> <span class="kd">var</span> <span class="nv">managedObjectModel</span><span class="p">:</span> <span class="bp">NSManagedObjectModel</span> <span class="p">=</span> <span class="p">{</span>
<span class="kd">let</span> <span class="nv">managedObjectModel</span> <span class="p">=</span> <span class="bp">NSManagedObjectModel</span><span class="p">.</span><span class="n">mergedModel</span><span class="p">(</span><span class="n">from</span><span class="p">:</span> <span class="p">[</span><span class="n">Bundle</span><span class="p">(</span><span class="k">for</span><span class="p">:</span> <span class="n">SQExceptionManager</span><span class="p">.</span><span class="kc">self</span><span class="p">)])</span><span class="o">!</span>
<span class="k">return</span> <span class="n">managedObjectModel</span>
<span class="p">}()</span>
</code></pre></div>
<p>Which is now working like a champ. Thanks to <code>rennarda</code> for their February answer of the
aforementioned SO post.</p>
<p>I believe that the piece that threw me a number of times was the reference for the class in
the <code>Bundle(for:)</code> call. For some reason, that regularly crashed the editor code and all
the other solutions I tried (like referencing the class directly) failed. In the end, it
seems as if the problem with referencing the class directly in that case had to do with something
inferring that I was trying to call an initializer instead of the class.</p>
So much LDAP, so little time2020-05-01T12:00:00-04:002020-05-01T12:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-05-01:/so-much-ldap-so-little-time.html<h2>The background</h2>
<p>Many years ago, all of my systems were pets. I tried to make them easier to manage by
standardizing on a single operating system (MacOS X Server at the time) and used management
tools that were part of that suite.</p>
<p>As time moved forward, Apple decided to concentrate …</p><h2>The background</h2>
<p>Many years ago, all of my systems were pets. I tried to make them easier to manage by
standardizing on a single operating system (MacOS X Server at the time) and used management
tools that were part of that suite.</p>
<p>As time moved forward, Apple decided to concentrate on the iPhone instead of the Xserve as
the next big thing and reduced their efforts on the server front. First, the hardware platform I was using
(Xserve) disappeared, and then MacOS X Server started taking big hits in functionality.</p>
<p>Meanwhile, Rob and I were moving our systems to be much more like cattle than pets. We
had standardized on SmartOS systems for running lightweight zones, and had standardized on
Ansible for reducing the overhead of rebuilding systems. In as many ways as we could,
all information on the servers, except for data which was rapidly changing or under user
control, was moved into a highly-reproducible configuration management system that allowed
us to try out new versions, run tests (some automated) and keep everything up to date by
nuking and paving the servers and rebuilding them from scratch each quarter or so.</p>
<p>The primary directory management system for MacOS X is LDAP. The implementation, called
Open Directory, was compatible with opensource and closed-source LDAP servers and scaled
pretty well in large environments. In small environments, the GUI was good enough that it
was not painful.</p>
<p>As we moved to SmartOS, though, we lost that built-in integration, and I had to create a
set of Ansible roles to connect SmartOS to LDAP in order to continue to use the same severs
(originally) and replicas of them based on OpenLDAP later. This all worked pretty well,
except that the LDAP servers themselves were pretty hand-tuned.</p>
<h2>State of affairs, pre-pandemic</h2>
<p>It took me a few years of spare time to get all of my systems under management, and the
last few systems to go were a set of LDAP servers that ran multiple domains worth of user
configuration for mail systems that I was running. These LDAP servers ran in an HA configuration
and generally worked really well. They also had securely hashed passwords, which I couldn't
un-hash and used an algorithm incompatible with our selected OS (this becomes important later).</p>
<p>As a part of my final push to get things under control, I decided to finally bite the bullet
and move the LDAP servers forward to the latest releases and get them on the automation
train. It required serious care to make sure that everything worked correctly, but our
habit of running separate instances for testing helped markedly in finding problems with
the system before taking it live.</p>
<p>I'll note here that I did have some other systems that served mail to other groups of users
that were built more recently. These used standard password hashes and were uncomplicated
by the use of LDAP.</p>
<h2>Quarantine Administration</h2>
<p>During the end of March and the beginning of April, I finally got the testing systems running
to my satisfaction. It seemed like the transition to production would be easy enough, since
the systems using the servers hardly ever changed, users generally weren't resetting their
passwords, and besides, there was no self-serve interface for that function anyway.</p>
<p>On the appointed day, I took one of the production servers offline (leaving it
in a dormant state that could be resuscitated quickly) and brought up one of the new servers
in the HA configuration, with the database of the previous production server. All seemed fine
on the servers I was testing on, but then I noticed a small number of users were having
trouble logging in.</p>
<p>I looked at the LDAP logs and there were no suspicious entries, but I noted in the database
itself that the users having trouble had no password entries whatsoever. I rolled back
the servers, but unfortunately the new data had been propagated to the other server.
Rebuilding from the most recent backup had the same problem... as far as I can tell, the
issue at this point was that a small number of users had something in their password records
that was failing the ldap data dump that I had used to reload things. Unfortunately, this
was the same dump I was using to rebuild the database, and that meant the old passwords
were effectively gone.</p>
<h2>The absence of complexity</h2>
<p>Although it's pretty clear that the problem was one of bad backups, I needed to get the
users back up and running, and I had no idea <em>why</em> the backups were bad. It might have been
something about the ancient MacOS X Server LDAP schema that I'd pulled forward, or some
change in the underlying configuration, but at this point, I didn't have time to figure it
out. I needed to get things back up and running.</p>
<p>Here is where I made a fine choice, with some prodding by Rob. Seeing that I was going to
need to reset passwords anyway for a number of users, I reached out to my user base and
requested new hashed passwords from them. But, since I was going to have to request these
passwords, I broadened the scope and got new hashes from everyone. This meant that I could
remove the complication of LDAP from my systems.</p>
<p>At the end of the day, I had everybody back up and running and a set of systems that are
even easier to operate than they were before.</p>
<p>I don't want anyone to take away from this that LDAP is bad. It isn't, it has places where
it's definitely the right solution. However, a small datacenter application with a slowly-changing
userbase and people with habits of good password hygiene is not one of those.</p>
<p>Let this serve as a reminder that we should always be open to making the right big changes
when given the opportunity.</p>
Welcome Bacula2020-03-31T13:46:00-04:002020-03-31T13:46:00-04:00Gaige B. Paulsentag:www.gaige.net,2020-03-31:/welcome-bacula.html<p>I wasn't originally going to write this up on the blog, but considering that we've just
finished our transition from our old backup software (BRU, no link) to <a href="https://bacula.org">Bacula</a>
community edition <em>and</em> considering that it's World Backup Day, it seemed like it would
make sense.</p>
<p>As many of you are …</p><p>I wasn't originally going to write this up on the blog, but considering that we've just
finished our transition from our old backup software (BRU, no link) to <a href="https://bacula.org">Bacula</a>
community edition <em>and</em> considering that it's World Backup Day, it seemed like it would
make sense.</p>
<p>As many of you are likely aware, ClueTrust hosts equipment at a top-tier datacenter for
providing services to our datacenter customers and our software customers alike.</p>
<h2>From Retrospect to BRU</h2>
<p>Since a lot of data that exists at the datacenter is not easily replaceable, we've had
on-site backups since early on in our operation of the racks of servers. At the time that
we started our backup journey, that was an LTO library hooked up to an Apple Xserve (RIP)
via Fibre Channel. Because of those particulars (Macintosh-based server, clients of a variety
of types, tape library), we had an extremely limited choice of backup software, and BRU
was basically it. At the time (December, 2006) we had been through a couple of tumultuous
years (literally, starting in December of 2003) evaluating various versions of BRU while
they got their MacOS X Server ducks in a row, and while Retrospect (our previous backup
software provider) didn't even seem interested in the Macintosh market at the time.</p>
<p>The journey with BRU was always a bit strained (I won't go into it here, but I believe
we're both happier to be out of that dysfunctional relationship). My expectations for
their responsiveness and customer-orientation were rarely met, and although there was a
lot of work on the MacOS X platform in 2003-2010, the release cadence for BRU Server from
that point seemed to grind to a near-halt. With that said, we have never lost a file with
BRU, the backups were always readable, and the format was simple enough that it gave
us confidence that even if an archive became corrupted, we could retrieve most of the data
from it.</p>
<h2>Taking it to the cloud</h2>
<p>By 2014, our preferred method of sending backups off-site no longer required me to take my
car to the datacenter and pull tapes out of the rack. Instead, we were moving to an
offsite storage mechanism that used "cloud storage". In our case, that meant <a href="https://aws.amazon.com/glacier/">AWS Glacier</a>.</p>
<p>There was no direct support for Glacier (or any other off-site backup mechanism) built
in to BRU, but they did have a disk-to-disk-to-tape model that could be run without the
"to-tape" part, which lead to my creating a bespoke Python solution for uploading our
archives to Glacier. I would not recommend that to most people, as the process is a bit
arduous and maintaining your own critical backup software is not recommended if you don't
have the discipline to regularly test it (especially when you don't control the server).</p>
<p>The solution we put together took advantage of the mostly self-contained nature of
of the BRU archives to shoot the data (encrypted after the fact, but otherwise unchanged)
to Glacier.</p>
<p>By 2015, as I mentioned in <a href="https://www.gaige.net/smartos-postfix-and-ipv6.html">SmartOS, Postfix and IPv6</a>,
we were in the process of shutting down our Xserves and replacing them with SmartOS.
Although BRU Client worked fine on the Solaris variant, we were never able to get the
licensing module to work with SmartOS, despite attempts to work with the BRU engineers.
As such, we ended up running our backup server in a sub-optimal configuration, a KVM-based
Ubuntu environment with a raw disk partition for scratch. Obviously, this would have been
much better if we'd been able to run on SmartOS with a LOFS partition directly taking advantage
of ZFS, but that was something we were never able to achieve.</p>
<p>Since certain catalog data wasn't readily extracted from the per-machine archives,
I re-engineered our custom solution in 2015 to make sure that we were storing
all of the salient metadata (type of backup, date, machine) in a way that would be more
easily addressed. This allowed us to find and remove old incrementals and so forth in
Glacier.</p>
<p>So, at this point, we had a custom off-site storage solution, hand-baked encryption, and
we were running in a KVM machine instead of running directly on the OS. Not an optimal
solution. Especially so when we had little hope that our chosen OS would be moving
forward in BRU-land. Things were working, but it required a lot of work to keep it up.</p>
<h2>Heading into the future</h2>
<p>As 2020 dawned, Rob and I are working on a number of datacenter initiatives, including
moving to a new SmartOS hardware platform and establishing beachheads in some other
locations. As part of this, I was looking to see what options we had for self-hosting
our off-site backups. Glacier wasn't hideously expensive (and its price pre byte decreases
occasionally), but if we're going to have off-site hardware, why not put our off-site
backups there.</p>
<p>The prospect of multiple sites also started me thinking about our current choice of a
commercial software solution. BRU wasn't unreasonably priced, but running a second server
would be a separate instance and that'd be a separate license. We could run the backups
over the internet from our other datacenter(s), but that would be a weird configuration and
likely not a performant one.</p>
<p>At this point, the idea hit that it was time to evaluate a solution that meets our 2020
needs, not our 2003 needs. As such, the requirements were:</p>
<ul>
<li>Open Source solution (if possible)</li>
<li>Support for a wide variety of OS, including SmartOS, Linux, macOS</li>
<li>Well-documented storage format</li>
<li>Classic Full, Incremental (optionally Differential) backups</li>
<li>Off-site cloud storage with compatibility with open-source storage solutions</li>
<li>Built-in public-private key encryption (preferably e2e from the device being backed up)</li>
<li>Built-in transport encryption and positive identification</li>
<li>Zero client trust required</li>
<li>Easy scripted installations of client and server</li>
<li>Flexible and scriptable configuration</li>
</ul>
<p>I looked around at a number of solutions, including the eventual winner, <a href="https://bacula.org">Bacula</a>,
and stalwarts such as <a href="http://amanda.org">Amanda</a>, as well as a ton of other, younger,
solutions. Many of the newer versions were either cloud-first or cloud-required, often they
trusted the client too much (such as handing the could credentials to the client), and
almost none of them had old-school multi-level backups, instead going for the much more
modern, Time-Machiney approach of a perpetually fresh backup.</p>
<p>I'm a big fan of Time Machine on macOS, but it's not the only backup I choose to use and
if I'm going to have a single backup mechanism, it's not going to be one where the loss of
some kind of long-term incrementally-updated database will result in sadness. As it stands
I've watched multiple times in the last decade as my Time Machine backups became corrupt
or needed to be moved off of older hardware. It's an extremely convenient capability, but
it's also brittle.</p>
<h2>The choice: Bacula</h2>
<p>So, after all that looking around, I turned my sights on Bacula as the leading contender.</p>
<ul>
<li>It has an open source version (yes, there have been some issues in the past with the
update cadence of the open source version, which lead to a fork named <a href="https://www.bareos.com">bareos</a>)</li>
<li>OpenSolaris is a supported OS, as are all of our other required OS</li>
<li>There is <a href="https://www.bacula.org/5.0.x-manuals/en/developers/developers/Overall_Storage_Format.html">storage format documentation</a></li>
<li>Backups are of the traditional Full, Incremental, Differential variety (although it also supports creating new synthetic Full backups)</li>
<li>Recent versions directly support S3-compatible off-site storage (including <a href="https://min.io">Minio</a>, and with Minio's help, <a href="https://backblaze.com">Backblaze</a>)</li>
<li>Encryption is end-to-end (except for attributes<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup> ) and uses public-private key encryption with optional multiple keys</li>
<li>TLS transport encryption and unique passwords for identifying each component to each other component
for positive identification</li>
<li>Clients are not in control and not allowed to contact the director directly</li>
<li>Installation from source, or a binary package (available for some platforms directly from
their website) is simple and easily scripted</li>
<li>Configuration parameters are all stored in text files which can be scripted easily</li>
</ul>
<p>All told, it hit all of our specifications and came in at a great price ($0), with available
commercial support if necessary and fully open source code.</p>
<p>Testing went well and I was able to script the building and packaging process as well as the
installation process on both the client and server end without difficulty.</p>
<p>In fact, one of the side-effects of a free solution is that we're now able to run a complete
test setup which mirrors our production setup and allows for easy validation of configuration
changes and upgrades.</p>
<p>There has been some difficulty with the built-in cloud support, but at least some of that
was owing to my problems getting the Minio-Backblaze gateway going. Now that's functional,
things seem to be working better. In addition, the mechanism for uploading data to Backblaze
(or S3, or Minio) is straightforward enough that uploading manually using <a href="https://rclone.org">rclone</a>
and downloading using the restore process in Bacula was completely successful.</p>
<p>By the way, performance has been excellent. It's not extremely fast when dealing with large
numbers of very small files (presumably file attribute overhead there), but it is highly
performant on large files and even the small file performance is acceptable. Because of the
text-based configuration, I've been able to do quite a bit of experimentation and our nightly
backup incremental across 18 different machines finishes in 8 minutes. Obviously, a full takes
substantially longer, but through the use of separate "tape changers" we're able to keep the
administratively-separate data separated while still running concurrent backups.</p>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>Bacula encrypts the data, but not attributes such as filenames, dates, modes, owners, etc.
Although contents of your backups are protected, frequently the metadata can be just as important
as the data in the file itself. As such, this begs for some kind of further encryption if you
are sending this data offsite for third-party storage. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
ssh key choices2020-03-07T16:36:00-05:002020-03-07T16:36:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-03-07:/ssh-key-choices.html<p>This weekend, Rob and I had been testing the use of hardware keys to secure ssh sessions,
especially for back-end console access and certain administrative functions. Since the
hardware keys are a special case, and cannot be added to the <code>ssh-agent</code>, we were slinging
around a fair number of command …</p><p>This weekend, Rob and I had been testing the use of hardware keys to secure ssh sessions,
especially for back-end console access and certain administrative functions. Since the
hardware keys are a special case, and cannot be added to the <code>ssh-agent</code>, we were slinging
around a fair number of command lines with <code>-i <keyfile></code> on them to point ssh at the key
we wanted it to offer. Also as part of the diagnostics, I was running with <code>-v</code>, so that
I could tell exactly what was going on. This is when I was reminded that ssh's choice of
keys isn't always what I expect (a problem I'd run into previously when maintaining keys
for a number of customer projects on the same device).</p>
<p>Without looking at the code, the observed key offer sequence appears to be:</p>
<ol>
<li>
<p>Keys added to your currently-available <em>ssh-agent</em> (unless you have disabled that with
<code>-o IdentitiesOnly=yes</code>)</p>
</li>
<li>
<p>Keys in your <code>~/.ssh/config</code> file referenced by the <code>IdentityFile</code> directive and matching
the host pattern (these are cumulative).</p>
</li>
<li>
<p>Keys specified with <code>-i</code> on the command line</p>
</li>
</ol>
<p>With that said, here are a few useful ssh-related notes:</p>
<ul>
<li>
<p>When you know you're going to need to use a password (as opposed to a key), use</p>
<div class="codehilite"><pre><span></span><code>ssh<span class="w"> </span>-o<span class="w"> </span><span class="nv">PubkeyAuthentication</span><span class="o">=</span>no
</code></pre></div>
</li>
</ul>
<p>which I have used <a href="https://www.keyboardmaestro.com/main/">Keyboard Maestro</a> to alias to
<code>sshP</code> in Terminal. This prevents running out of login attempts before you get a chance
to enter the password, as most ssh servers will only allow 3-5 attempts,
including pubkeys and interactive passwords.</p>
<ul>
<li>
<p>If you need to prevent your config file from being used, <code>-F /dev/null</code> will override
your config file with an empty one.</p>
</li>
<li>
<p>Of course, you can always use <code>ssh-add -D</code> to remove all keys from the <em>ssh-agent</em>, but
that affects all terminal sessions on your machine.
As an alternative, you can avoid consulting the <em>ssh-agent</em> by unsetting the shell variable
<code>SSH_AUTH_SOCK</code>, which is used to locate the authentication socket. Since this is a shell
variable, it only affects the shell that you perform it in, so it leaves your other
terminal windows able to use the agent's keys.</p>
</li>
<li>
<p>None of the identity commands affect the operation of the <code>-A</code> command line switch (or
the corresponding <code>ForwardAgent yes</code> directive in your config file). So, even if you
use <code>-o IdentitiesOnly=yes</code> to keep the session initiation from offering the keys in
the agent at the time, an <code>-A</code> flag on the command line will allow you to use the keys
on further communication from the target host (useful for things like bastion hosts, such
as the ones we're securing).</p>
</li>
</ul>
Update to nginx_alias_map2020-02-26T11:47:00-05:002020-02-26T11:47:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-02-26:/update-to-nginx_alias_map.html<p>I've been doing a bunch of maintenance on my two blogs (company and personal) and one
purpose has been to track down malformed and mis-mapped URLs on the site. Since both
have been through changes in the underlying blog engine a couple of times, there are
multiple sets of URLs …</p><p>I've been doing a bunch of maintenance on my two blogs (company and personal) and one
purpose has been to track down malformed and mis-mapped URLs on the site. Since both
have been through changes in the underlying blog engine a couple of times, there are
multiple sets of URLs that point to the same content. Generally, since there was a long
period of time between each of the respins, the search engines picked up the changes in
URLs, but occasionally I will see log entries for the two-engines-ago format, and I'd like
to fix those as well, especially since the same content is still on the site in most
cases.</p>
<p>The most recent versions (the <a href="https://blog.cartographica.com">Cartographica blog</a> was on
SquareSpace and <a href="https://gaige.net">Gaige's Pages</a> as in drupal) are already mapped and
had simple mappings due to good URI choices. However, both of these blogs were previously
(initially) in <a href="https://www.geeklog.net">Geeklog</a>, and it had a format that was based on
the <code>article.php</code> file and a query string.</p>
<p>As mentioned in my <a href="https://www.gaige.net/pelican-plugin-for-nginx-redirection.html">nginx_alias_maps</a> post
last year (this week, it turns out), I had written a bit of code to produce nginx maps to
handle redirections.</p>
<p>However, as written, the <code>map</code> that I was generating was using the <code>$uri</code> variable in
nginx, which cooks the URI by removing things like the query string. Obviously, this won't
work for the old Geeklog query string-based redirection, so I needed to move to using the
full <code>$request_uri</code>. That was fine, but came with drawbacks as well. As I mentioned, the
<code>$uri</code> variable is cooked by nginx, removing relative directory traversals, double-slashes,
query strings, etc. For most of my URIs, this is a much better fit. As such, I decided
to complicate the plugin a bit and add support specifically for URIs which contained a <code>?</code>
as an indicator of query strings, and to process them in a second stage map. It's a little
more time consuming, although it's not noticeable on my blogs.</p>
<p>The solution was to run the <code>$uri</code> map for any URIs not containing query strings and then
run the <code>$request_uri</code> map for any URIs that did contain them. So, if you had an alias
entry such as the one for
<a href="https://www.gaige.net/load-up-those-album-covers.html">Load up those album covers</a> (header shown here):</p>
<div class="codehilite"><pre><span></span><code><span class="n">Date</span><span class="o">:</span><span class="w"> </span><span class="mi">2003</span><span class="o">-</span><span class="mi">04</span><span class="o">-</span><span class="mi">29</span><span class="w"> </span><span class="mi">11</span><span class="o">:</span><span class="mi">41</span>
<span class="n">Alias</span><span class="o">:</span><span class="w"> </span><span class="sr">/node/4921,/</span><span class="n">article</span><span class="o">.</span><span class="na">php</span><span class="o">?</span><span class="n">story</span><span class="o">=</span><span class="mi">2003042913413622</span>
<span class="n">Tags</span><span class="o">:</span>
<span class="n">Category</span><span class="o">:</span><span class="w"> </span><span class="n">macintosh</span>
<span class="n">Title</span><span class="o">:</span><span class="w"> </span><span class="n">Load</span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="n">those</span><span class="w"> </span><span class="n">album</span><span class="w"> </span><span class="n">covers</span>
</code></pre></div>
<p>the code will generate entries in two maps:</p>
<div class="codehilite"><pre><span></span><code><span class="k">map</span><span class="w"> </span><span class="nv">$uri</span><span class="w"> </span><span class="nv">$redirect_uri_1</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">~^/node/4921$</span><span class="w"> </span><span class="s">https://</span><span class="nv">$server_name/load-up-those-album-covers.html</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">map</span><span class="w"> </span><span class="nv">$request_uri</span><span class="w"> </span><span class="nv">$redirect_uri</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">default</span><span class="w"> </span><span class="nv">$redirect_uri_1</span><span class="p">;</span>
<span class="w"> </span><span class="kn">~^/article\.php\?story=2003042913413622$</span><span class="w"> </span><span class="s">https://</span><span class="nv">$server_name/load-up-those-album-covers.html</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Note here that the first map maps to <code>$redirect_uri_1</code> and the second one maps to
<code>$redirect_uri</code>, with a default value of <code>$redirect_uri_1</code>. Because of the way that
nginx evaluates maps, you can't use <code>$redirect_uri</code> in both cases.</p>
<p>As with previous versions, you need to include the map in your <code>http</code> stanza in your nginx
configuration, and you also need to check the value of <code>$redirect_uri</code> and send it back
as a redirect if present:</p>
<div class="codehilite"><pre><span></span><code><span class="k">include</span><span class="w"> </span><span class="n">/opt/web/output/alias_map.txt</span>;
<span class="k">server</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="s">*:80</span><span class="w"> </span><span class="s">ssl</span><span class="p">;</span>
<span class="w"> </span><span class="kn">server_name</span><span class="w"> </span><span class="s">example.server</span><span class="p">;</span>
<span class="w"> </span><span class="c1"># Redirection logic</span>
<span class="w"> </span><span class="kn">if</span><span class="w"> </span><span class="s">(</span><span class="w"> </span><span class="nv">$redirect_uri</span><span class="w"> </span><span class="s">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">return</span><span class="w"> </span><span class="mi">301</span><span class="w"> </span><span class="nv">$redirect_uri</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="kn">location</span><span class="w"> </span><span class="s">/</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kn">alias</span><span class="w"> </span><span class="s">/opt/web/output</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Of course, if you only have one or the other type of redirection, the code will make sure
to only create a single-stage map.</p>
<p>Updated code is now available as
<a href="https://github.com/gaige/nginx_alias_map">nginx_alias_map</a> on <a href="https://github.com">github</a>.</p>
Client Certs and Intermediate CAs2020-02-24T10:12:00-05:002020-02-24T10:12:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-02-24:/client-certs-and-intermediate-cas.html<h1>Why client certificates?</h1>
<p>RS wrote about <a href="https://technotes.seastrom.com/2016/11/01/preventing-drivebys-with-client-certs.html">Preventing drive-bys with client certs</a>
and although we'd discussed this method for some time, I hadn't deployed it yet. However,
some recent log-spelunking had led me to determine that I liked the idea of a second
layer of protection on some of my sites …</p><h1>Why client certificates?</h1>
<p>RS wrote about <a href="https://technotes.seastrom.com/2016/11/01/preventing-drivebys-with-client-certs.html">Preventing drive-bys with client certs</a>
and although we'd discussed this method for some time, I hadn't deployed it yet. However,
some recent log-spelunking had led me to determine that I liked the idea of a second
layer of protection on some of my sites.</p>
<p>Just for clarification, this is being implemented as a first-layer of authentication and
only will grant access to the basic functionality of the server. There will be further
login requirements beyond that. However, as Rob notes in his post, there is significant
advantage to keeping the wild internet away from the login screen (or anything else they
might be able to exploit). So, as Rob says, this is drive-by prevention.</p>
<p>Previous to this exercise, I'd occasionally used nginx's built in authentication to require
another user name and password before accessing the site, but this is a bit tedious due to the
way that browser prompts and password systems (such as 1Password) work. Client-side certificates
don't have this problem, at least on the Apple ecosystem, that authorization is tied to the
device auth.</p>
<p>(TL;DR: put all predecessors of the client CA in the CRL)</p>
<h1>Intermediate CAs</h1>
<p>As RS's article is sufficient to get you going, why is this article necessary? Because I
have a tendency to take opportunities like this to explore unnecessarily complex models,
such that I can understand how the internals work and can employ them as needed in the future.</p>
<p>In this particular case, I have run an internal CA now for over a decade. This is used only
on internal communications, although before <a href="https://letsencrypt.org">Let's Encrypt</a>, it was
also used for securing web sites that weren't going to be accessed by people outside of the
organization (and family and friends). With LE dropping the marginal cost of certificates
to zero (and the fixed cost to the automation that RS and I have already created), the
private CA hasn't been getting a lot of use.</p>
<p>A few years back, my original CA cert expired and it caused a notable amount of pain (this
was still before LE was in general release), mostly because I needed to send out new certificates
to each of my users and make sure they installed them on all of their various devices.
In order to head this off for the future, I wanted a long-running CA certificate, something
with an expiration beyond the date of my likely retirement, so something in the 2040's...</p>
<p>In the "real world" Root CAs have frequently had this kind of duration. For example the DigiCert
High Assurance EV Root CA expires in 2031, and was originally issued in 2006; so, basically
25 years. However, they also have significant security on their Root CAs (for example, physical
HSMs holding the keys) and do not issue normal certificates directly from the Root, but instead
issue from intermediate CAs that have much shorter expirations. So, I figured I could approximate
this using a secure USB storage device (tamper-proof and encrypted with a long PIN) for my
Root, and by issuing certificates off of intermediates with much more limited lifespans.</p>
<p>For the few servers I've used this on, it has worked well. I trust the Root certificate on
each of my devices and then the web servers send the intermediate (signed by the root) and
the server certificate along with it.</p>
<h1>Intermediate CAs and debugging client-side certificates</h1>
<p>Since I already have the intermediate CAs (it turned out I'd created one for client certificates
and another for server certificates originally), it seemed like an easy enough exercise to
take RS's recipe and apply it to my client certs. I generated a new private key and a new
client certificate for myself and then went about the configuration.</p>
<p>Simple enough, I took the intermediate CA and uploaded it to my server, along with the
current Certificate Revocation List (CRL) and placed them into the <code>ssl_client_certificate</code>
and <code>ssl_crl</code> stanzas respectively, making sure to turn <code>ssl_verify_client on;</code> as well.</p>
<p>That didn't work. No good error message, just an SSL error from both Safari and Chrome.
I should note here that debugging client certificates requires quitting your browser frequently.
If you don't there's often some piece of state that will either create false negatives or
false positives. So, while testing, basically quit your browser and re-start it between
every attempt. However, I've found no befit to restarting the machine.</p>
<p>I tried adding the Root CA to the Intermediate certificate in <code>ssl_client_certificate</code>, but
this was unnecessary (as it should be if the client contains an authorized copy of the root,
and the server is explicitly instructed to use the intermediate certificate as the base for
authorization).</p>
<p>I'll note here that adding <code>debug</code> to the end of my nginx config's <code>error_log</code> line
would have been helpful at this point. There were definitely errors occurring and only the
server knew what they were.</p>
<p>Once I turned on debug, it was clear that the CRL verification was causing difficulty and
so I validated that by commenting the <code>ssl_crl</code> line in the nginx config and restarting the
server and my browser and things worked.</p>
<p>No CRL is probably a bad idea, so I did some looking around and it became clear that nginx
wanted to check both the CRL of the Root and the Intermediate, so I concatenated the
two CRL files and uploaded them to the server, and now the server is working fine.</p>
<p>In retrospect it made sense. If you're going to ask for client certificate verification and
you're going to provide a CRL, you should provide CRLs all the way to the root.</p>
Larry Tesler at NCSA2020-02-19T13:40:00-05:002020-02-19T13:40:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-02-19:/larry-tesler-at-ncsa.html<p>Today I read of the passing of <a href="https://en.wikipedia.org/wiki/Larry_Tesler">Larry Tesler</a>,
a computer scientist with a long and storied career, spanning Xerox PARC, Apple, Amazon,
Yahoo, and others. He's considered the father of the modeless interaction model (think
Cut/Copy/Paste on the Mac).</p>
<p>I met Larry in the late 1980s, when …</p><p>Today I read of the passing of <a href="https://en.wikipedia.org/wiki/Larry_Tesler">Larry Tesler</a>,
a computer scientist with a long and storied career, spanning Xerox PARC, Apple, Amazon,
Yahoo, and others. He's considered the father of the modeless interaction model (think
Cut/Copy/Paste on the Mac).</p>
<p>I met Larry in the late 1980s, when I was at NCSA and he was visiting there on sabbatical
from Apple, with his wife (a geophysicist with a time grant on our supercomputer). As the
head Mac enthusiast at the facility (ok, maybe Brand Fortner had that title at the time, but I
was Mac developer and he was more management), one might think that Larry and I would hit
it off. And, perhaps we would have, if it hadn't been for the Mac II Ethernet Card.</p>
<p>Back in those days, NCSA had a great relationship with Apple. We had a lab full of Macs to
go with our Cray, and we liked to talk about them. In 1986, we'd released NCSA Telnet, my
first serious professional software and a key communications package for the Mac in those
days. Also, we'd been part of the educational beta test for the Macintosh II<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>.</p>
<p>When Larry arrived with his wife to begin their work at NCSA, he brought with him a brand
new Mac II with an even newer Ethernet card. This card was so new, they were not available
to the general public, or even most of folks who were part of the beta test, instead we were
still using LocalTalk<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup> for our networking.</p>
<p>Larry comes in and assembles his Mac II, complete with test Ethernet card and I'm brought in
to hook up the network connection at his desk. Introductions are made, although I had no
idea who he was, and he certainly didn't know me from Adam. I hooked up the thin-net connection
to the back of the Mac II, pulled out a trusty floppy to load the latest NCSA Telnet onto
the machine, and sat down to get things working.</p>
<p>At the time, I don't recall if we had any other systems that had Ethernet adapters, but there
were some units available for the Mac SE with the Processor Direct Slot, and I believe I'd
already gotten the Ethernet drivers working there. Regardless of the timeline, we did manage
to get the networking working quickly, and NCSA Telnet was humming along... for a few minutes
and then everything came to a halt. The Mac II had crashed and there was no clear cause.</p>
<p>Being two software guys, Larry and I both assumed that there was something wrong with NCSA
Telnet and its interaction with the Ethernet driver. I came back in and sat down to take a
further look at what was going on. Debugging tools on the Mac weren't great in those days, but
there was always Macsbug, and if time was right, maybe TMON.</p>
<p>After trying to figure it out for a while, we pulled the plug on the Ethernet card and
tried NCSA Telnet over LocalTalk. That worked fine, and would allow him to get to work if
he wanted to. Meanwhile, I had access to the machine in off-hours to try and figure the
problem out. I came back to the office later in the day and tried again. It took much
longer to reproduce the problem, but it eventually came back. I restarted again and let the
system just sit while I went to talk to Tim or Tom, or one of the other folks on the team to
see if they had any bright ideas. Nothing new from them, but when we came back, the system
had crashed. The odd part was that NCSA Telnet wasn't loaded and, in those days, you only had
one application running at a time. Further, NCSA Telnet didn't use any resident drivers
as it had full access to the hardware when it was running. Curious...</p>
<p>The next day, I got a frantic call that the machine was acting up, even on LocalTalk. As
far as Larry was concerned, this was proof positive that the problem was with NCSA Telnet and
not with the Ethernet card. I went over to check out the problem and noticed that, unlike
the previous day, we'd left the Ethernet card plugged in. I unplugged it and we tried to
reproduce the problem, and could not. The problem was now coming into focus.</p>
<p>A few days later, some of these cards had made it to other universities in the test program
and I was getting email telling me that they were experiencing crashing problem. Of course,
it looked like NCSA Telnet might be involved, since it was one of the communications programs
that would use Ethernet. However, in these cases as well, the problems also seemed to
occasionally manifest when NCSA Telnet wasn't running.</p>
<p>Eventually (not sure if this was with a week or two), we were able to put the pieces together
and realize that high traffic on the network caused the Ethernet hardware to freeze up,
causing the Mac to follow suit. Apparently, Apple's test labs were much more orderly and
had a lot fewer collisions, whereas our crowded educational networks were collision-fests and
this was directly related to the problems on the card. A re-spin of the hardware was in
order and it wasn't too long before we had working Ethernet cards in the Mac II that could
handle a network with real activity on it.</p>
<p>Unfortunately, I never really got to know Larry, although we exchanged pleasantries while he
was visiting, he was thoroughly involved in the work on geophysics at the time. Plus, I don't
think that the Ethernet card situation resulted in a lot of good will...</p>
<p>Interestingly, while doing some searching for this piece, I ran across Kent Beck's
<a href="https://medium.com/@kentbeck_7670/larry-tesler-1945-2020-b910429f12eb">Piece on Larry Tesler</a>
and if you notice about half-way down the page, you'll get to an anecdote about a geological
core sample browser, that was what he was working on.</p>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>As an aside, we had to keep the Mac II in a locked room, and agree to a number
of limitations on use, including a specific prohibition on taking off the cover. A bit
disappointing, since it had 6 NuBus slots. However, when it arrived, it had no cover. Technically,
we were never in violation. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
<li id="fn2" class="footnote-item"><p>I'm not sure if this was before or after they renamed AppleTalk to LocalTalk,
but since AppleTalk eventually became the name of the protocol stack and LocalTalk the name
of the medium, I have disambiguated the two here. <a href="#fnref2" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
Overwatch Leaves nVidia's GeForce Now2020-02-12T09:11:00-05:002020-02-12T09:11:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-02-12:/overwatch-leaves-nvidias-geforce-now.html<p>According to an <a href="https://www.pcworld.com/article/3526458/activision-blizzard-just-pulled-its-games-from-nvidias-geforce-now.html">article</a>
on <a href="https://www.pcworld.com">PCWorld</a>, <a href="https://www.blizzard.com/en-us/">Activision-Blizard</a> has pulled
all of their titles from <a href="https://www.nvidia.com/en-us/">nVidia's</a>
<a href="https://www.nvidia.com/en-us/geforce-now/">GeForce Now</a>.</p>
<p>In my days as CTO of <a href="https://haste.net">Haste</a> (a service that improves network connections for gamers),
I had occasion to spend a fair amount of time playing <a href="https://us.shop.battle.net/en-us/product/overwatch?p=20991">Overwatch</a> as part of
our test regime …</p><p>According to an <a href="https://www.pcworld.com/article/3526458/activision-blizzard-just-pulled-its-games-from-nvidias-geforce-now.html">article</a>
on <a href="https://www.pcworld.com">PCWorld</a>, <a href="https://www.blizzard.com/en-us/">Activision-Blizard</a> has pulled
all of their titles from <a href="https://www.nvidia.com/en-us/">nVidia's</a>
<a href="https://www.nvidia.com/en-us/geforce-now/">GeForce Now</a>.</p>
<p>In my days as CTO of <a href="https://haste.net">Haste</a> (a service that improves network connections for gamers),
I had occasion to spend a fair amount of time playing <a href="https://us.shop.battle.net/en-us/product/overwatch?p=20991">Overwatch</a> as part of
our test regime. As a Mac user (and the only one on our technical team), I had a much more
difficult time than my co-workers getting things to run smoothly and beautifully. While Richard and Taric were
playing on their huge honking Razer and SurfaceBooks with high-end nVidia graphics, I had to
settle for running games (via <a href="https://support.apple.com/boot-camp">Bootcamp</a>) on my
then current MacBook Pro. I had the highest-end graphics available in it, but the Radeon Pro
at the time was no match for the then-current GeForce, especially given the thermal constraints
of the 15" MacBook Pro at the time.</p>
<p>Having done quite a bit of team-based FPS back in the <a href="https://en.wikipedia.org/wiki/InterCon_Systems_Corporation">InterCon</a>
days (shout out to <a href="https://www.bungie.net">Bungie's</a> early history as a Mac games company,
and the awesome game, <a href="http://marathon.bungie.org">Marathon</a>), I needed to see if I could
reduce the technical discrepancy between my rig and my fellow players. So, I proceeded to
do the only thing a laptop Mac user could do, short of buying a separate PC gaming rig, and
I acquired an AKiTiO (now <a href="https://macsales.com">OWC</a>) <a href="https://eshop.macsales.com/item/AKiTiO/NODET3IAO/">Node</a>
eGPU box, capable of running a modern, beefy GPU connected to my Mac. Since I was not trying
to use my internal Retina display and I wasn't trying to use it under MacOS, I didn't have
any significant problems getting things set up, and now I was cooking with gas with an
external 4K monitor that I could drive at max resolution and 60Hz.</p>
<p>Of course, those of you who know me know this didn't help my gameplay that much, but at least
I no longer felt left behind and there wasn't any real doubt that my skill was showing through
in the game.</p>
<p>But, in the end, the system was a bit fiddly, not very portable, and required shutting down
from MacOS and rebooting into Windows, which I needed to keep up-to-date in order to play
the game. Not a great experience.</p>
<p>When I returned to DC and stepped back to being an advisor at Haste, I sold off my GeForce
card (they depreciate quickly), mothballed the AKiTiO and stopped playing Overwatch.</p>
<p>Fast forward a couple of years, and nVidia announces that they're finally going live with
GeForce now and not only is it reasonably priced, but there would be a free tier. With that,
I couldn't put off giving it a try. Honestly, the gameplay blew me away. Considering that
I was playing without having to leave macOS or reboot, or add any more hardware to my machine,
this was pretty awesome. I signed up, and jumped back in to Overwatch, enjoying myself for
the better part of an hour before calling it quits for the day.</p>
<p>Unfortunately, that looks like it has quickly come to an end, because of Blizzard's actions,
and now I will have to decide if there's another game that's worth taking up. But, at least
I no longer feel like I'll be left out when my friends are discussing some great new game
which isn't (yet) available on the Mac.</p>
Developing on a 2019 Mac Pro2020-01-08T14:55:00-05:002020-01-08T14:55:00-05:00Gaige B. Paulsentag:www.gaige.net,2020-01-08:/developing-on-a-2019-mac-pro.html<p>There's been a lot of discussion about the 2019 Mac Pro and various assertions that it's over-designed,
overpriced, or underpowered. Since I decided to replace my venerable 2013 Mac Pro<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>
with a 2019 Mac Pro, I figured I'd write up my experience with the device as a developer.</p>
<h2>The …</h2><p>There's been a lot of discussion about the 2019 Mac Pro and various assertions that it's over-designed,
overpriced, or underpowered. Since I decided to replace my venerable 2013 Mac Pro<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>
with a 2019 Mac Pro, I figured I'd write up my experience with the device as a developer.</p>
<h2>The Codebase</h2>
<p>So that people have context for what the codebase is that I'm talking about here, I currently
ship 3 "products":</p>
<ul>
<li><a href="https://macgis.com">Cartographica</a> (Macintosh GIS product)</li>
<li><a href="https://cartomobile.com">CartoMobile</a> (Mobile GIS for field data entry - iOS)</li>
<li><a href="https://loadmytracks.com">LoadMyTracks</a> (Free utility for working with GPS devices on macOS)</li>
</ul>
<p>Cartographica is by far and away the largest of these three projects. Besides over 250K lines
that I have written over the years (CLOC lines of code, not counting whitespace or comments),
there are another approximately 1.2M lines of third-party code<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup> that gets compiled every time I
do a Clean and Build.</p>
<p>LoadMyTracks is much smaller, about 44K lines of code by me (some shared with Cartographica,
but for the purposes of this article, I'm interested in describing what's compiled) and about
37K lines of third-party code.</p>
<p>CartoMobile is in the middle, with approximately 134K lines written mostly by me (the remainder of
the ClueTrust code was written by a contractor early on in CartoMobile's life) and about
380K lines of third-party code.</p>
<p>Beyond the code, we have a fair number of tests that run in the system as well. Cartographica has
on the order of 3500, CartoMobile has around 700, and LoadMyTracks around 150. These are
each mostly unit tests, with some integration tests as well. The UI tests (a combination of
manual and automated) are not included here, as they don't run automatically in the CI environment.</p>
<h2>Build Environment</h2>
<p>I try to keep on the latest build environments wherever possible, which means a fair
amount of work maintaining the code base (and removing deprecations). Thus, the build environment
is Xcode 11.3 as of the arrival of the 2019 Mac Pro. I'm using Xcode's parallel testing, so
it's not out of the ordinary for the 2019 Mac Pro to have 8 or 10 copies of Cartographica
running for the in-process tests and a similar number of <code>xctest</code> instances running during
the stand-alone tests.</p>
<p>For Continuous Integration (CI), I run <a href="https://jenkins.io">Jenkins</a> on a set of Mac Minis, with most of the build
work being done by a 2018 Mac Mini. CI builds are run using <a href="https://fastlane.tools">fastlane</a> and Jenkins pipelines.</p>
<h2>Before and after</h2>
<h3>Cartographica</h3>
<p>Now that I've set out the environment and the challenge, let's see how the Mac Pro did with it.
Clean building on the 2013 Mac Pro (6-core) takes 358s, or close to 6 minutes. Clean building
on the 2019 Mac Pro (16-core) takes 82s. For the record, building in the Xcode GUI directly on the 8-core
2018 Mac Mini takes 177s.</p>
<p>The tests themselves are a different matter. The Mac Mini is still beating the 2019 by about
18s of wallclock time (131s vs 110s), mostly owing to 2 single-threaded tests (run in parallel) that take
51s on the Mac Pro and only 31s on the Mac Mini. I'm tracking this issue down, but in this case, the
Mac Mini is running 10.14 still and the 2019 (like it's predecessor) is running 10.15.2.
Since these long-running tests involve spawning a subprocess to run a shell script which executes a unix-level
executable approximately 400 times, I'm guessing this is related to a difference in the
shell execution. For reference, the 2013 Mac Pro took over 180s to run these tests. Interestingly,
the MacBook Pro (running 10.15.2) runs the tests similarly to the Mac Mini, taking 35s to
run the long shell-script based test.</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th style="text-align:right">Build Time</th>
<th style="text-align:right">Test Time</th>
<th style="text-align:right">Scripted Test</th>
</tr>
</thead>
<tbody>
<tr>
<td>2019 Mac Pro (16 core Xeon)</td>
<td style="text-align:right">82s</td>
<td style="text-align:right">131s</td>
<td style="text-align:right">51s</td>
</tr>
<tr>
<td>2019 MacBook Pro (8 core i9)</td>
<td style="text-align:right">169s</td>
<td style="text-align:right">116s</td>
<td style="text-align:right">35s</td>
</tr>
<tr>
<td>2018 Mac Mini (6 core i7)</td>
<td style="text-align:right">177s</td>
<td style="text-align:right">110s</td>
<td style="text-align:right">31s</td>
</tr>
<tr>
<td>2013 Mac Pro (6 core Xeon)</td>
<td style="text-align:right">358s</td>
<td style="text-align:right">223s</td>
<td style="text-align:right">>180s</td>
</tr>
</tbody>
</table>
<h3>LoadMyTracks</h3>
<p>Differences on LoadMyTracks were much smaller, which I assume is because the code base is so much
smaller. The test times are almost identical.</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th style="text-align:right">Build Time</th>
<th style="text-align:right">Test Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>2019 Mac Pro (16 core Xeon)</td>
<td style="text-align:right">16.5s</td>
<td style="text-align:right">3s</td>
</tr>
<tr>
<td>2019 MacBook Pro (8 core i9)</td>
<td style="text-align:right">18.8s</td>
<td style="text-align:right">3s</td>
</tr>
<tr>
<td>2018 Mac Mini (6 core i7)</td>
<td style="text-align:right">15.5s</td>
<td style="text-align:right">3s</td>
</tr>
<tr>
<td>2013 Mac Pro (6 core Xeon)</td>
<td style="text-align:right">43.1s</td>
<td style="text-align:right">4s</td>
</tr>
</tbody>
</table>
<h3>CartoMobile</h3>
<p>CartoMobile's larger codebase (thus more amenable to compiling in parallel), results in another
significant win for the 2019 Mac Pro over the 2013 (and the Macbook Pro this time). As with
Cartographica, the Mac Mini is a close 3rd behind the 2019 MacBook Pro</p>
<table>
<thead>
<tr>
<th>Machine</th>
<th style="text-align:right">Build Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>2019 Mac Pro (16 core Xeon)</td>
<td style="text-align:right">20.8s</td>
</tr>
<tr>
<td>2019 MacBook Pro (8 core i9)</td>
<td style="text-align:right">32.1s</td>
</tr>
<tr>
<td>2018 Mac Mini (6 core i7)</td>
<td style="text-align:right">47.2s</td>
</tr>
<tr>
<td>2013 Mac Pro (6 core Xeon)</td>
<td style="text-align:right">48.5s</td>
</tr>
</tbody>
</table>
<h2>Conclusion</h2>
<p>Not surprisingly, it looks like the larger the codebase, and the more amenable to being
built or tested in parallel, the better the 2019 Mac Pro runs. This was pretty much the case
with the previous Mac Pro as well, when compared to the Mac Minis and laptops at the time.</p>
<p>I don't have an iMac Pro to perform these tests on, but based on the significant performance
improvements over the 6- and 8-core machines, I expect I'd want the 14- or 18-core iMac Pro
to try and reach the same performance. However, those two machines are clocked at 2.5 and 2.3Ghz
respectively, which is significantly lower than the 2019 Mac Pro 16-core's 3.2Ghz.</p>
<p>Assuming that the 18-core would be sufficient, a similarly configured machine (2TB NVMe SSD, default
video card, 64GB of RAM) would run me $8278 list. The 2019 Mac Pro configuration I purchased,
along with 64GB of 3rd Party RAM and LG 5K display, listed at $8799 + $1299 + $442 = $10,540.
It's definitely $2,262 more, although if I were purchasing the iMac, I'd probably upgrade the
video card, since I don't have any option to do it later (reducing the difference by $700).
In addition, if it hadn't been for the strange spot that the 2013 Mac Pro existed in
(Thunderbolt 2 not quite having enough bandwidth for 5K 60Hz on a single cable), I wouldn't
need to replace my Dell 5K monitor that I'd used with it for the last 5 years, which would
drop the difference by another $1299.</p>
<p>With that said, for compiling my main App, the processor speed reduction of close to
30% might be an issue.</p>
<p>In the end, the value proposition of a Mac Pro vs. competing platforms
is a judgement call for anyone who is considering it. The combination
of the flexibility (I changed graphics cards and expanded RAM in my previous "cheese grater"
models) and the external monitor to me makes enough of a difference to come down on the side
of the Mac Pro. Assuming that Apple keeps up with it, I may end up replacing the computer
and keeping the components, such as upgraded graphics cards and maybe RAM. If they don't
keep up with the upgrades, the replaceable CPU is likely to result in opportunities to improve
performance, as had been done with previous models containing replaceable CPUs.</p>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>Seriously, I was pretty happy with it, despite the clear lack of upgradability and limits to expansion. <a href="#fnref1" class="footnote-backref">↩︎</a></p>
</li>
<li id="fn2" class="footnote-item"><p>Shout out here to the most significant third-party code that we use: <a href="https://gdal.org">GDAL</a>,
<a href="https://sparkle-project.org">Sparkle</a>, and <a href="https://proj.org">Proj</a>. <a href="#fnref2" class="footnote-backref">↩︎</a></p>
</li>
</ol>
</section>
gitignore as a service2019-12-09T13:02:00-05:002019-12-09T13:02:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-12-09:/gitignore-as-a-service.html<p>When you're looking to quickly create an appropriate .gitignore file for a new repository,
you can save yourself some time, and possibly aggravation, by using <a href="https://gitignore.io">gitignore.io</a>.</p>
<p>Available as either a website with a very simple interface (and completion), or as a simple
API-based service <a href="https://docs.gitignore.io/install/command-line">documentation for the API</a>
and …</p><p>When you're looking to quickly create an appropriate .gitignore file for a new repository,
you can save yourself some time, and possibly aggravation, by using <a href="https://gitignore.io">gitignore.io</a>.</p>
<p>Available as either a website with a very simple interface (and completion), or as a simple
API-based service <a href="https://docs.gitignore.io/install/command-line">documentation for the API</a>
and how to call it from the command line is available from the site.</p>
<p>Source code is available on GitHub, and the license is a nice MIT license.</p>
<p>I've generally found the information to be pretty exhaustive, although occasionally you can
run into policy decisions, so I find it useful to grab a copy and then review it. Remember,
if you miss a file that you want to have saved, it may just be left out of your repository.</p>
<p>One additional note: if you like the default for a set of items you generated from the
web site, the top of the generated file contains the API URL to pull it in the future.</p>
<p>All told, a really useful service, thanks to those who created and have contributed to this!</p>
Ansible become: useful and dangerous2019-11-26T20:51:00-05:002019-11-26T20:51:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-11-26:/ansible-become-useful-and-dangerous.html<p>OK, now that I have your attention with the catchy title, let me get right into the
reason behind this post. Rob has been doing a lot of work lately on a set of roles
to provision raspberry pi systems. I'm grateful for the work in this area, because
frankly …</p><p>OK, now that I have your attention with the catchy title, let me get right into the
reason behind this post. Rob has been doing a lot of work lately on a set of roles
to provision raspberry pi systems. I'm grateful for the work in this area, because
frankly, I find them a bit annoying to get boot-strapped. Although we're not ready to
fully publish the workflow (I expect that'll happen in due time and likely on Rob's
fine <a href="https://technotes.seastrom.com">Technotes</a> blog).</p>
<h2>About become</h2>
<p><code>become</code> and it's associates (<code>become_user</code>, <code>become_method</code>, and the least-used <code>become_flags</code>)
provide a way to handle privilege escalation in situations where you otherwise are logging
in to a system with an unprivileged user. In our particular context, the default configuration
on a raspberry pi has a single default user who can <code>sudo</code>, but can't execute privileged
commands natively.</p>
<p>By setting <code>become: true</code>, privilege execution will by default become user <code>root</code> using
the method <code>sudo</code>. Clearly, <code>become_user</code> and <code>become_method</code> provide a way to modify who
and how you become.</p>
<h2>A warning about over-scoped <code>become</code></h2>
<p>In cases like the raspberry pi, it's tempting to just run all commands using become, and
there's a seemingly convenient command-line flag to do just this, <code>-b</code>. By asserting
<code>-b</code>, you tell ansible to execute all commands with escalated privilege.</p>
<p>Note that I said <em>all commands</em>? This <em>includes</em> commands that are executed locally using
ansible's <code>local_action</code> command. My initial reaction to this was surprise, figuring that
<code>local_action</code>, as opposed to <code>delegate_to: localhost</code> would likely avoid the <code>become</code>
capability, but in retrospect, it's a good bit of consistency.</p>
<p>So, don't over-become, and don't use the <code>-b</code> command-line flag. If you have a host, such
as the aforementioned pi, that you need to <code>become: true</code> for all commands, you can
assign <code>ansible_become=yes</code> using your configuration method of choice (stick it in your
inventory file, add it to <code>group_vars/all.yml</code>, or maybe even in a specific subgroup if
you have a mix of machines that have varying <code>become</code> needs).</p>
<h2>Other interesting uses for become</h2>
<p><code>become</code> is a really useful feature, and we regularly use it in plays for command-specific
one-offs, such as using <code>become: yes</code> and <code>become_user: postgres</code> to run PostgreSQL commands
with the benefit of being the <code>postgres</code> user. Or with live content systems like django,
I use <code>become_user: www</code> to run some of the maintenance scripts so that we don't have to
manually re-chown all of the files that are collected for static web service.</p>
<h2>Summary</h2>
<p>All told, <code>become</code> is really useful, but pay attention to where you're using it, and just
avoid entirely using <code>-b</code>, because you never really know what role might include some
<code>local_action</code> that has a bug and might execute <code>rm -rf /</code> for you...</p>
NetNewsWire rises again!2019-11-11T19:17:00-05:002019-11-11T19:17:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-11-11:/netnewswire-rises-again.html<p>One of the very first posts on this blog (16 years ago in the beginning of 2003) was
entitled <a href="https://www.gaige.net/all-of-your-favorite-sites-at-a-glance.html">All of your favorite sites at a glance</a>
which discussed a new pair of apps (<a href="https://ranchero.com/netnewswire/">NetNewsWire</a>
and NetNewsWire Lite) that I'd just started using.</p>
<p>Considering the fallout for RSS from the …</p><p>One of the very first posts on this blog (16 years ago in the beginning of 2003) was
entitled <a href="https://www.gaige.net/all-of-your-favorite-sites-at-a-glance.html">All of your favorite sites at a glance</a>
which discussed a new pair of apps (<a href="https://ranchero.com/netnewswire/">NetNewsWire</a>
and NetNewsWire Lite) that I'd just started using.</p>
<p>Considering the fallout for RSS from the Google Reader shutdown in 2013 and all that involved,
it is surprising, but heartening that those of us who remain independent bloggers have
clung to the RSS standard for making it easy to keep up with sites we're interested in.</p>
<p>I wanted to take a few minutes to add my congratulations to <a href="https://inessential.com/">Brent Simmons</a>,
who has himself been blogging for over 20 years, and add my thanks to Brent and all of the
contributors who made the new version of NetNewsWire possible. I'm now back to using it
and it's the best experience I've had reading RSS on my Mac in years (well, since the last
great NetNewsWire).</p>
Separating Ansible roles for fun and profit2019-11-11T13:41:00-05:002019-11-11T13:41:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-11-11:/separating-ansible-roles-for-fun-and-profit.html<p>At ClueTrust, we use a lot of automation to run our systems. It's mostly how just a
couple of us can manage hundreds of virtual servers and keep them up-to-date and
operational.</p>
<p>A few years back, I moved from using <a href="https://puppet.org">Puppet</a> to
<a href="https://ansible.org">Ansible</a>, mostly at the suggestion of RS, who …</p><p>At ClueTrust, we use a lot of automation to run our systems. It's mostly how just a
couple of us can manage hundreds of virtual servers and keep them up-to-date and
operational.</p>
<p>A few years back, I moved from using <a href="https://puppet.org">Puppet</a> to
<a href="https://ansible.org">Ansible</a>, mostly at the suggestion of RS, who was finding Ansible
a solid choice for both network and server automation.</p>
<p>At a high level, my problems with Puppet were that my code tended to rot pretty quickly
due to changing dependencies, and the Ruby-based DSLs were overly complex and hard to
debug a few months after using them last.</p>
<p>Ansible is based on Python (a language I'm very comfortable with), and although there's a
bit less elegance in an ordered series of steps in a configuration file (a playbook in ansible),
there's a whole lot less difficulty in figuring out what the system is doing.</p>
<p>After years of working with Puppet, I wanted to move to deliberately move to Ansible in
a way that would maximize ease of re-use in similar situations. This lead me to the
following guiding principles:</p>
<ol>
<li>
<p>Scope and general nature of Ansible code should get more specialized and opinionated
as it moves from commands to roles to plays to playbooks to environments.</p>
</li>
<li>
<p>At every level, use generalized code from the previous level when reasonable.</p>
</li>
<li>
<p>As you move to more specialized code, the code becomes less useful to other groups.
By the time you reach a playbook, the code should mostly be tying together reusable lower-
level modules (especially roles and commands) and applying inventory data to them.</p>
</li>
<li>
<p>Wherever possible, parameterize items that are easy to parameterize. Especially when
writing roles, use of parameters that can be defaulted or automatically set in the role
can make it much easier to expand the use of a role across operating systems. In our case
although most of our roles are aimed at SmartOS, there are a growing number that can also
be applied to Linux environments when necessary.</p>
</li>
<li>
<p>Discrete units of code should be treated as separate and therefore stored in their own
repositories. For example, since Roles are expected to be highly reusable, they're always
each stored in their own individual git repository and included using Galaxy-style
<code>requirements.yml</code> files. (Note: ansible-galaxy can point at arbitrary git and hg repositories
as well as the marketplace. This includes private git repos, which is exactly what we do.
The format is <code>- src: git+user@server.fqdn/repo</code>)
Similarly, separating environments containing playbooks into separate repos also provides
advantages.</p>
</li>
</ol>
<h2>Why so many Repos?</h2>
<p>Looking at number 5 above might make you think that we have a lot of repositories for our
ansible roles, and you'd be right.
Not only does this promote reuse across the organization, but it also provides an easy
point at which to make a role public if appropriate. Further, eventually you may need to
make a breaking change in a role. If you do this in a separate repo, you can just set the
specific version in the <code>requirements.yml</code> file for playbooks that don't need, or can't
use the newer version just yet. However, this can be done on a role-by-role basis.
In contrast, using a single omnibus repository for all roles will make this difficult if
you need different versions of 2 roles in a single playbook.</p>
<h2>Ansible environments</h2>
<p>Similarly, when we build playbooks, we tend to put them in what I call <code>environments</code>.
An ansible environment is a location that contains playbooks, inventories, and requirements.
Because they contain inventories, they also frequently contain customization data related
to similar hosts. Although these can be administrative domains, they're commonly also
functional. However, due to the level of reuse of roles, the environments can be split in
any way that makes sense. At a minimum, the environment contains:</p>
<ul>
<li>at least one inventory</li>
<li>at least one playbook</li>
<li>a <code>requirements.yml</code> file</li>
<li>various vars directories (<code>group_vars</code> and <code>host_vars</code>)</li>
<li>various files directories (<code>files</code> and <code>templates</code> are well known, but we also use
<code>host_files</code> for common host-specific files)</li>
<li>a README.md file</li>
</ul>
<p>It's not uncommon for us to begin with an environment that contains one or more
simple playbooks for related systems, and then evolve those playbooks over time into roles.
This is effectively the next obvious step in refactoring ansible for us. If inside of
an environment a series of steps becomes so common that it exists across many playbooks,
it likely is moved to an included play. From there, if it's usable across other environments
it is then turned into a role and gets its own repository. In this way, we can control the
complexity of the playbooks and environments and standardize our roles to minimize
configuration drift between similar systems.</p>
<h2>Stage vs Prod</h2>
<p>Everyone has their own way of doing inventories, but since I'm discussing our environments
here, I figured I'd also touch on how I do inventories to manage stage vs production. For
the most part, I tend to work with 1:1 stage and production systems. This way, whenever we
need to validate a specific system, we have a way to do so without having to cobble something
together by hand. Not that I necessarily test every individual host every time, but by
keeping the mechanism standardized, it's easy to do so when necessary.</p>
<p>Generally speaking, I have 2 inventories in each environment, <code>prod</code> and <code>stage</code>. By specifying
these on the command line explicitly (<code>-i prod</code> or <code>-i stage</code>), it is always clear whether
you're going to affect crucial systems or not. At the base of these inventories, I include
every host in a group named for inventory. As such, we can use <code>stage.yml</code> and <code>prod.yml</code>
in the <code>group_vars</code> directory to specify items that are specific to the two environments.</p>
<p>Since <code>all.yml</code> in <code>group_vars</code> will be at the bottom of the priority, it's easy to do
things like temporarily change the location or version of a binary in stage by putting
the stage value in <code>stage.yml</code>. However, it's generally safer to define all of the vars
that might be shared in <code>all.yml</code>. Obvious exceptions to this are items that, if shared,
would cause problems, such as database server addresses. In these cases, put those
explicitly only in the <code>stage.yml</code> and <code>prod.yml</code>, so that they're undefined if left out.</p>
<h2>Host-specific files</h2>
<p>For the most part, it makes sense that an environment's key configuration files would be
in <code>files</code> or <code>templates</code>, and that items like nginx configurations for specific hosts or
classes of system would be called out at the top level and configured using variables.</p>
<p>However, there are cases where we either use roles to auto-generate files or have files
in a particular environment that are always different for each machine in the environment
and might be too cumbersome to put in the configuration. Especially in the first case,
we need to be able to check for existence quickly, and to create or modify the data as
necessary, without affecting any other configuration parameters. In this case, we use
our host-specific files directory <code>host_files</code>. Accessed as <code>host_specific_files</code> in
<code>all.yml</code>, it is defined thusly:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">host_specific_files</span><span class="p">:</span><span class="w"> </span><span class="s">"{{</span><span class="nv"> </span><span class="s">inventory_dir</span><span class="nv"> </span><span class="s">}}/host_files/{{</span><span class="nv"> </span><span class="s">inventory_hostname</span><span class="nv"> </span><span class="s">}}"</span>
</code></pre></div>
<p>And, thus for every individual inventory host, it specifies a particular directory in the
environment (relative to the inventory). Files can then be placed inside this directory
(or in sub-directories) that are created the first time a host is provisioned and saved
when the environment is committed to the repository. Obviously, these should only be
public files unless they are encrypted using ansible-vault. However, in cases of some
persistent private keys for machines or services, we will frequently encrypt those and
store them in the repo, with the encryption keys kept locally on the user's machine.</p>
<p>Note: this isn't for high-security items. Obviously, those should be much more limited in
access. However, this mechanism it does provide for a good way to limit re-provisioning
of certain keys that might otherwise cause people to become complacent about seeing
changes frequently. For example, ssh host keys.</p>
SSH probe bad behaviors and the sshd settings that make them worse2019-08-26T11:49:00-04:002019-08-26T11:49:00-04:00Gaige B. Paulsentag:www.gaige.net,2019-08-26:/ssh-probe-bad-behaviors-and-the-sshd-settings-that-make-them-worse.html<p>Over the past few weeks, RS and I have noticed an increasing number of unexplained
failures logging in over SSH with both manual and automated means. On most of the servers
it was just an inconvenience, but as it started to become more frequent, it became a
significant issue for …</p><p>Over the past few weeks, RS and I have noticed an increasing number of unexplained
failures logging in over SSH with both manual and automated means. On most of the servers
it was just an inconvenience, but as it started to become more frequent, it became a
significant issue for some of our automation scripts.</p>
<p>However, the most significant effects were to our git server which only allows access
over ssh. Not surprisingly, this server gets a frequent, short-duration connections from
our CI server as well as individual git clients that are checking for the status against
their remotes. As such, the sudden spate of failures caused us to look into the issue.</p>
<p>Upon looking at the machine in question, it was clear that the usual ssh background probing
was going on, but that the bot that was probing us was getting confused and leaving the
connection open. This was causing a build-up of <code>ESTABLISHED</code> connections which was running
up against the default limit of 10 on SmartOS. Individually, these were not coming in very
quickly (probably in order to not trigger banning software), but since they were taking
a very long time to transition from <code>ESTABLISHED</code> to <code>CLOSED</code>, they were taking up space
in the table of 10 and causing additional inbound connections (from our legitimate users)
to be unceremoniously shut down.</p>
<p>Upon further investigation of the <code>sshd_config</code> file, I noted that the <code>LoginGraceTime</code>
in SmartOS is set to a default value of 600. Each connection had 10 minutes to wait
around until a successful login occurred or until it was disconnected. That seems a bit
long even if you're allowing password authentication, but in our key-only environment it
is extreme, and so we cut it back to 8.</p>
<p>Interestingly, this caused a fair number of connections to be stuck in <code>FIN_WAIT_1</code>,
an indication that the bot that was probing us was just terminating the connection on its
end and moving on, without sending any indication to our side. That might just be a crash
on their side, or it might be a tuned bot. The distinction would be hard to make, but
fortunately, the sshd process quits immediately upon closing its side of the connection
due to <code>LoginGraceTime</code> expiring, so the result is a sufficient reduction in the number
of outstanding sshd's in <code>ESTABLISHED</code>, which fixed our immediate problem.</p>
<p>It was a bit of a surprise this hasn't bit us on other machines, but considering that most
of our automations completely destroy and rebuild the VM over sshd, the likelihood that
a significant number of abandoned connections build up during the build process is pretty
low.</p>
<p>We will be updating our standard sshd configuration to take care of this in the next
build cycle and I've updated machines which have seen this problem repeatedly on an
individual basis.</p>
Git subtrees for Perforce users2019-08-18T08:00:00-04:002019-08-18T08:00:00-04:00Gaige B. Paulsentag:www.gaige.net,2019-08-18:/git-subtrees-for-perforce-users.html<p>For many years, I was a happy <a href="https://perforce.com">Perforce</a> user. Despite
clearly not fitting their precise model, I had a three-user license which allowed
me and my bots to appropriately work on my code base. I have a number of pretty
complex projects, which often have overlapping code and I took …</p><p>For many years, I was a happy <a href="https://perforce.com">Perforce</a> user. Despite
clearly not fitting their precise model, I had a three-user license which allowed
me and my bots to appropriately work on my code base. I have a number of pretty
complex projects, which often have overlapping code and I took advantage of their
evolving code sharing mechanisms. Initially by using a single repository with
<em>workspaces</em> that included code from different locations, and then moving to
the more powerful (and, in my mind, more easily understood) streams paradigm.</p>
<h2>Moving slowly to Git</h2>
<p>I generally follow an active trunk (master) strategy where new development
is done on the trunk and branches are used to pull off releases. Generally
speaking, I tend to develop like I've got a team even when it's just me.</p>
<p>As Perforce evolved, they tried to expand their offerings to include more git-like
capabilities, in particular having local copies of your repositories. Obivously,
this carried very similar benefits and disadvantages of git, but since lightweight
branching and offline use were becoming more important, the perforce model needed
to adapt.</p>
<p>It was through this lens that I started to move my repositories over to git.
Frankly, I'd come up with adaptations which allowed me to make use of a central
server system without compromising mobility (in particular, for years my primary
server ran on my laptop, with a mirror running on my CI server), but over time
the effort to maintain the synchronized servers became more significant. Perforce
has a nice git front end for their server (git-fusion) which allows you to read
and write Perforce repos using a git interface, giving you most of the
advantages of both systems, but it falls down right where I tended to need it,
on handling incorporated streams. The git submodule overlay was tedious and
finicky at best and eventually convinced me to move fully to git.</p>
<p>At some other point in time, I'll discuss <a href="https://gitolite.com">gitolite</a>,
which is what I'm using as a git server right now, but that won't be important
for the rest of this story.</p>
<h2>Adoption of git in my complex code base</h2>
<p>Initially, I thought I was going to be able to use git-fusion and submodules
to pull my Perforce streams out of the repositories and keep things in sync,
but that proved problematic. Maybe it was the number of files or interesting
merge tactics I'd experimented with, but in the end, I found that my large,
complex repositories were best exported without Perforce's help with the
submodules. The result, though, was that I had huge, monolithic, git repositories
each of which contained complete histories of all of my submodules.</p>
<p>You'll note that I'm already using the word 'submodule' here, and that's not
an accident. I tend to develop my shared code modules with their own projects,
their own tests, and separate versioning and branching. With Perforce, this
minimized the amount of duplication in the code base, and kept the system clean.
Further, it lead to my creation of a CI environment that would test all of my
modules individually as well as inside the larger projects. All told, I'm fond
of it.</p>
<p>So, as I moved to a git lifestyle, I followed instructions by others on how
to split my repositories using <code>git filter-branch</code> and successfully teased out the
submodules that I shared between my major projects. It took a bit of trial and
error, but it also proved a great experiment into the general forgiveness of git.
All told, not having to commit to a central server makes it a lot easier to
verify your full operation before you make a big mistake.</p>
<h2>Git submodules and subtrees</h2>
<p>When I started working my code into git, I'd read a number of pieces on the
various advantages and disadvantages of git submodules and git subtrees. The
basic feel of most of these articles was "Why you should never use [insert the
other technique here]". Generally speaking, I didn't see a lot of benefit to the
use of subtrees. The vast majority of the complaints seemed to be that submodules
were a pain (they can be complex), and they're difficult to deal with if you
need to make a lot of changes to other people's code. In my case, my submodules
were almost exclusively internal. Thus, the issue with OPC wasn't a big deal.</p>
<p>Until I got to a library that I like to festoon with my own framework. In particular,
I'm talking about the venerable <a href="https://gdal.org">GDAL</a>, a widely-used library
for raster and vector I/O used by much of the GIS industry. My macOS framework
is a pretty large and complex beast, with multiple subcomponents (GDAL has a
variety of optional libraries) and some private modifications. When using it
in Perforce, I'd used a multi-stream system: an <code>//import/gdal</code> repo that
carried an exact copy of the GDAL source and a <code>//GDALFramework</code> stream that
was used to build the framework. The latter repo was imported into my
application workspace using the stream functions. So, I'd grab the latest GDAL
(which at the beginning was using svn) and then check that in to my <code>//import/gdal</code>
repo, which I'd then merge into my <code>//GDALFramework</code> stream and fix any
requirements or compilation/test errors, then I'd check that in and update the
stream in my main application. It wasn't horrific, but getting to the point
where I could run my application tests against it was a long track, so I didn't
update as frequently as I'd like.</p>
<p>After I moved to git, my framework remained an intrinsic part of my source, along
with its included <code>gdal</code> library. That worked fine, as long as I didn't need to
update anything, but the cross-stream import branch information was long out of
date, and I couldn't reasonably import a new version of GDAL without a lot of
care. Enter git subtrees.</p>
<p>I had considered using GDAL as a git submodule and just forking it for the
few changes I would need to make. That would work reasonably well, in theory,
but the build environment that I use isn't the same as the standard environment
and that meant that I couldn't reasonably test the code before committing into
master, which I deem a no-no (ok, I could, but it would mean coordinating separate
submodules with a set of special branches, which is do-able, but a pain in the
neck). By using git subtrees instead, the imported code
remains tightly coupled with the surrounding code, but isolated into an
appropriate subdirectory. I can still submit changes back to the GDAL repo when
appropriate, but I also get to fully test the code before I put it onto my
master branch.</p>
<p>Subtrees are easy to work with, especially in comparison to submodules. As long
as you're willing to commit to forward momentum in lock-step with your updated
subtrees, the mechanism works great. Add a subtree in a subdirectory:</p>
<div class="codehilite"><pre><span></span><code><span class="go">git subtree add --prefix GDALFramework/gdal gdal v2.4.2 --squash</span>
</code></pre></div>
<p>and you've got the code you want, right where you asked for it. Explaining that
line a bit, it adds the code from the <code>gdal</code> origin (which I added
using <code>git remote add</code>),
from the <code>v2.4.2</code> tag in the subdirectory <code>GDALFramework/gdal</code> and uses the
<code>--squash</code> option to limit the commits brought into the local repo. Once that's
done, you can make changes to your heart's content, and merge in changes from
the original repo by doing:</p>
<div class="codehilite"><pre><span></span><code><span class="go">git subtree pull --prefix GDALFramework/gdal gdal <Branch id> --squash</span>
</code></pre></div>
<p>where <code><branch id></code> is whatever branch or commit id you want to update to. Do
your tests, make your changes, verify that everything is working and you're good.
Commit your changes when you're ready.</p>
<p>When you've got something that you want to commit back to the community, you
will need to fork the original repository and push to that origin so that you
can prepare your changes for assimilation:</p>
<div class="codehilite"><pre><span></span><code><span class="go">git subtree push --prefix=GDALFramework/gdal mygdalfork master</span>
</code></pre></div>
<p>I'm happy with my decision to use a combination of submodules and subtrees. I'm
sure that either method could be used for the purposes I'm using the other method
for, but I find the distinction is useful. In particular, I can easily experiment
with code branches between my apps using submodules, and I can work easily with
code from others which needs some adaptation using subtrees.</p>
Pelican plugin for NGINX redirection2019-02-20T09:32:00-05:002019-02-20T09:32:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-02-20:/pelican-plugin-for-nginx-redirection.html<p>When I set out to move Gaige's Pages to a static web generator, chronicled in
<a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">Gaige's Pages moves to static generation</a>,
I stated one of the reasons that I favored <a href="https://github.com/getpelican/pelican">Pelican</a>
was because it is written in <code>python</code>, which is a language that I'm intimately familiar
with.</p>
<p>Not surprisingly, that …</p><p>When I set out to move Gaige's Pages to a static web generator, chronicled in
<a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">Gaige's Pages moves to static generation</a>,
I stated one of the reasons that I favored <a href="https://github.com/getpelican/pelican">Pelican</a>
was because it is written in <code>python</code>, which is a language that I'm intimately familiar
with.</p>
<p>Not surprisingly, that decision because useful pretty quickly. As I was working on moving
the <a href="https://blog.cartographica.com">Cartographica Blog</a> from SquareSpace to Pelican, I
was having some concern over the redirection method I used in Gaige's Pages, the
<a href="https://github.com/Nitron/pelican-alias"><code>pelican-alias</code></a> plugin.</p>
<p>The <code>pelican-alias</code> plugin is a highly useful piece of code, especially if you're going to
be placing your pelican site on a server you don't control. The method of redirection is
to place a file at the original location and then redirect using HTML. This is effective
without propagating multiple copies of your pages in multiple locations (as would be the
case if you used a mapping in your web server), however it has two undesirable effects:</p>
<ol>
<li>It causes a slight browser-induced delay for the HTML reload command to be recognized
and executed</li>
<li>It doesn't tell search engines to permanently relocate your pages to their new location</li>
</ol>
<p>I realized that the problem I was looking to solve was slightly different. In my case, I have
complete control over the web server (<a href="http://nginx.org">nginx</a> in my case),
and therefore can provide configuration information to it directly,
including having it redirect using HTTP 301 and 302 codes.</p>
<p>Furthermore, since I have a fine static blog engine with plugin support written in a language that
I am comfortable with, and with plenty of example code, I was able to pull together a pretty
simple plugin to create a map from the <code>alias</code> attribute in my blog postings to the final
published URI. The result is now available as
<a href="https://github.com/gaige/nginx_alias_map">nginx_alias_map</a> on <a href="https://github.com">github</a>.</p>
<p>I'm now running two sites using it and both seem to be performing admirably.</p>
<p>Code is published under the MIT license and pull requests are welcome.</p>
XCUITests and macOS2019-02-05T14:23:00-05:002019-02-05T14:23:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-02-05:/xcuitests-and-macos.html<p>A number of years ago, I set out to automate a set of manual tests that we've been
using for years to validate functionality and UI in Cartographica. I've been through
a lot of technologies over the years, some expensive commercial tools, some open
source technologies. I won't go through …</p><p>A number of years ago, I set out to automate a set of manual tests that we've been
using for years to validate functionality and UI in Cartographica. I've been through
a lot of technologies over the years, some expensive commercial tools, some open
source technologies. I won't go through an exhaustive list of what we used and
how we found them, but I will say that the last technology we used was <a href="https://appium.io">Appium</a>,
a tool created mostly for mobile (macOS does benefit from its younger, more
popular sibling occasionally) which models itself on <a href="https://www.seleniumhq.org">Selenium</a>
and uses that popular web testing tool as the orchestration layer.</p>
<p>When I moved to using Appium, I'd already been doing something similar using
and hand-modified version of <a href="https://github.com/pyatom/pyatom">Pyatom</a>, a python-based automated
testing framework for the Mac that uses the macOS Accessibility Framework to provide testing. I
implemented some custom modifications for testing Cartographica, but the system was required that all
tests be written manually in python. That's fine, I'm good with python, but there were no recording
mechanisms or other tools to make the process easier.</p>
<p>After happening upon Appium when trying to automate testing for CartoMobile, I realized I could take
advantage of the Selenium offshoot to get some much-needed tooling around my testing for Cartographica.
This worked pretty well, although the recording tools never turned out to be a good shortcut, so I
ended up writing all of my tests in Python manually anyway. Still, there were third-party tools, and
integrations with popular testing and continuous integration frameworks, such as Jenkins.</p>
<p>In the end, I added about 25 tests for Cartographica using Appium before the relative brittleness got
to me. In addition to being a bit difficult to manage on the Mac, I ran into some significant timing
problems and issues with item occlusion when doing drag testing. Although that wasn't the end of the
world, it was a large time sync and the tests themselves, even when hardened had a failure rate of 3-8%
per run, which meant they weren't so much a gate as a hurdle. My last check-in notes when I was still
hoping for better GUI test development were in 2014, and read "Try once again to get the drag-and-drop
tests working under 10.10". They did not after many days, and I finally gave up and went back to
manually running and validating the</p>
Dynamic XCTests2019-01-20T22:04:00-05:002019-01-20T22:04:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-01-20:/dynamic-xctests.html<p>For a number of years, Cartographica has had a lot of tests-on the order of 1500+, but a few of them are
quite a bit bigger than they should have been, owing mostly to their data-driven nature.</p>
<p>This post describes the method used to provide dynamic test creation for Cartographica …</p><p>For a number of years, Cartographica has had a lot of tests-on the order of 1500+, but a few of them are
quite a bit bigger than they should have been, owing mostly to their data-driven nature.</p>
<p>This post describes the method used to provide dynamic test creation for Cartographica.</p>
<h2>Cartographica File Formats</h2>
<p>Cartographica uses a couple of big libraries to provide a wide variety of file import and export capabilities,
a situation which is not uncommon in the GIS world, as most of the players, including ESRI use (and make available)
libraries for accessing file formats that they either create or need access to. The result is that many applications
have access to a large number of file formats.</p>
<p>In order to deal with this large variety of formats in an intentional way, I created an internal catalog that is used
by Cartographica to handle most operations with file formats, including:</p>
<ul>
<li>User-readable format names</li>
<li>System-usable identifiers</li>
<li>Format limitations</li>
<li>Available test files and verification data</li>
<li>Identification of implementation source</li>
<li>Identification of licensing</li>
</ul>
<p>The files that contain this information are shipped as an internal part of the shipping code, and are part of the
source bundle used to create the software.</p>
<p>As such, the test information in the file format data file provides essential information for programmatically testing
Cartographica's import and export capabilities (including round-trip testing where applicable).</p>
<h2>Implementing Dynamic Testing</h2>
<p>Historically, Cartographica's file format testing appeared in the test system as a small set of tests which ran for
a long time and provided little debugging information when a specific file format test failed. In addition, due to
the data-driven nature, it was difficult to test just a single file format.</p>
<p>Over time, I considered a number of different ways to create dynamic tests, most of which required code generation,
which was creating too much complexity for me to approve of. I needed a solution that would work within the confines
of our Continuous Integration build system (Jenkins) and would create obvious signals if the tests either failed or
were not executed for some reason.</p>
<p>This weekend, I took another stab at handling the problem and was successful. The mechanism is straight-forward,
almost elegant, and very Objective-C.</p>
<p>After choosing to spend some time in this area, I read up on the official mechanism for creating dynamic tests, using
<code>defaultTestSuite</code>, which looked like it would be a good option, except that it has a side-effect of not playing
well with the IDE for re-running tests, and I really wanted to be able to do that. Because the suite creation code
is only called when running the whole suite, it doesn't make available the individual tests if they are re-executed.</p>
<p>The examples I'd found also resulted in a large number of identically-named methods being called, which was pretty useless
in terms of providing visibility to the tests. The solution to this problem was to create separate implementations
using the dynamic runtime properties of Objective-C. By doing so, the second problem (identical naming) was solved and
I could create uniquely-named, highly-identifiable test names.</p>
<p>As an added bonus, this also paved the way to fixing my first complaint through the removal of code. By creating all
of my tests with names starting with <code>test</code>, the system will auto-discover the tests (as long as they are created
in the <code>+initialize</code> method). Therefore, I was able to remove the code which otherwise created the default TestSuite,
since the Objective-C introspection for methods beginning with <code>test</code> would find all of my new dynamic tests.</p>
<p>I had help from a couple of sites in putting this approach together, and the result has been great. My 2 file format
tests are now approximately 200, and they are individually named for what they do and to what data types.</p>
<h2>In-depth Code Review</h2>
<p>The particular code that I'm using works with hundreds of different file formats, and most of them have
sample data files and information that allows me to verify that they're all loading correctly. To index
all of this information, I use a strongly-typed XML Schema for these descriptions. The strong typing
facilitates the strong validation of the files. Since all of this information (including driver names)
is in this XML file, the best way to truly test this data is by running automated tests that interpret the
files.</p>
<p>The actual code implementation is relatively simple:</p>
<ul>
<li><code>+initialize</code> method to parse the file and set up the tests</li>
<li><code>+CTUTaddInstanceMethodWithSelectorName</code> used to register a block for the method</li>
</ul>
<p>Our test registration code calls the add instance method with the block that executes the actual tests.
The parameter passed to the block (<code>ImportExportFileTests</code>) is just a subclass of <code>XCTestCase</code>, as that's
what <code>XCTestCase</code> expects to call <code>test*</code> methods with.</p>
<p>In this particular example code, we're passing in a <code>NSString</code> containing an XML fragment which contains
information on the individual test files. The <code>-runFileTestWithFileNode</code> method referred to inside of
the block is a basic test routine which I was able to reuse without modification. In this case, I chose
to place run all the tests for a file format in the same test.</p>
<p>Looking at the code below, <code>checkTestFileList</code> verifies that at least one test file exists for this file
format and driver (and the level of test that we're doing... some tests take a long time and are only
executed when <code>isExhaustive</code> is set).</p>
<p><code>fileFormatName</code> is the human-readable name of the format, <code>fileFormatType</code> is the UTI for the file type,
some of which are vendor-specific, others are defined in Cartographica. For the UTIs we define, we create
associations for <code>com.ClueTrust.Cartographica.external.<type_name></code>, where <code><type_name></code> is a unique
identifier for each type. Since that's a large, mostly useless, prefix and we're guaranteed that it's
unique (since we check that elsewhere when validating the file format file), we shorten it to <code>external</code>.
Finally, <code>.</code> is replaced with <code>_</code> to comply with method name requirements.</p>
<p>The block passed to <code>+CTUTaddInstanceMethodWithSelectorName:block:</code> executes the tests themselves, using
the standard <code>XCTest</code> assertions to flag problems.</p>
<div class="codehilite"><pre><span></span><code><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">testMethodBaseName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">@"testVectorImport_"</span><span class="p">;</span>
<span class="bp">NSArray</span><span class="w"> </span><span class="o">*</span><span class="n">checkTestFileList</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">fileFormat</span><span class="w"> </span><span class="n">nodesForXPath</span><span class="o">:</span><span class="w"> </span><span class="n">testFileSearchString</span><span class="w"> </span><span class="n">error</span><span class="o">:</span><span class="nb">nil</span><span class="p">];</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">checkTestFileList</span><span class="p">.</span><span class="n">count</span><span class="o"><</span><span class="mi">1</span><span class="p">)</span>
<span class="w"> </span><span class="k">continue</span><span class="p">;</span>
<span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">fileFormatName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[[</span><span class="n">fileFormat</span><span class="w"> </span><span class="n">attributeForName</span><span class="o">:</span><span class="w"> </span><span class="s">@"name"</span><span class="p">]</span><span class="w"> </span><span class="n">stringValue</span><span class="p">];</span>
<span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">fileFormatType</span><span class="w"> </span><span class="o">=</span><span class="p">[[[[</span><span class="n">fileFormat</span><span class="w"> </span><span class="n">attributeForName</span><span class="o">:</span><span class="s">@"typeID"</span><span class="p">]</span><span class="w"> </span><span class="n">stringValue</span><span class="p">]</span>
<span class="w"> </span><span class="nl">stringByReplacingOccurrencesOfString</span><span class="p">:</span><span class="s">@"com.ClueTrust.Cartographica.external"</span><span class="w"> </span><span class="n">withString</span><span class="o">:</span><span class="w"> </span><span class="s">@"external"</span><span class="p">]</span>
<span class="w"> </span><span class="nl">stringByReplacingOccurrencesOfString</span><span class="p">:</span><span class="w"> </span><span class="s">@"."</span><span class="w"> </span><span class="n">withString</span><span class="o">:</span><span class="s">@"_"</span><span class="p">];</span>
<span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">testName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">testMethodBaseName</span><span class="w"> </span><span class="n">stringByAppendingString</span><span class="o">:</span><span class="w"> </span><span class="n">fileFormatType</span><span class="p">];</span>
<span class="n">NSXMLDocument</span><span class="w"> </span><span class="o">*</span><span class="n">doc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">NSXMLDocument</span><span class="w"> </span><span class="n">documentWithRootElement</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="n">fileFormat</span><span class="w"> </span><span class="k">copy</span><span class="p">]];</span>
<span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="n">xmlString</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">doc</span><span class="w"> </span><span class="n">XMLString</span><span class="p">];</span>
<span class="p">[</span><span class="nb">self</span><span class="w"> </span><span class="n">CTUTaddInstanceMethodWithSelectorName</span><span class="o">:</span><span class="w"> </span><span class="n">testName</span><span class="w"> </span><span class="n">block</span><span class="o">:^</span><span class="p">(</span><span class="n">ImportExportFileTests</span><span class="w"> </span><span class="o">*</span><span class="n">test</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">NSXMLDocument</span><span class="w"> </span><span class="o">*</span><span class="n">xml</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[[</span><span class="n">NSXMLDocument</span><span class="w"> </span><span class="n">alloc</span><span class="p">]</span><span class="w"> </span><span class="n">initWithXMLString</span><span class="o">:</span><span class="w"> </span><span class="n">xmlString</span><span class="w"> </span><span class="n">options</span><span class="o">:</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="n">error</span><span class="o">:</span><span class="nb">nil</span><span class="p">];</span>
<span class="w"> </span><span class="n">NSAssert</span><span class="p">(</span><span class="w"> </span><span class="n">xml</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need XML"</span><span class="p">);</span>
<span class="w"> </span><span class="bp">NSArray</span><span class="w"> </span><span class="o">*</span><span class="n">testFileList</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">xml</span><span class="p">.</span><span class="n">rootElement</span><span class="w"> </span><span class="n">nodesForXPath</span><span class="o">:</span><span class="w"> </span><span class="n">testFileSearchString</span><span class="w"> </span><span class="n">error</span><span class="o">:</span><span class="nb">nil</span><span class="p">];</span>
<span class="w"> </span><span class="n">NSAssert</span><span class="p">(</span><span class="w"> </span><span class="n">testFileList</span><span class="p">,</span><span class="w"> </span><span class="s">@"no test files for %@"</span><span class="p">,</span><span class="w"> </span><span class="n">fileFormatName</span><span class="p">);</span>
<span class="w"> </span><span class="n">NSAssert</span><span class="p">(</span><span class="w"> </span><span class="n">testFileList</span><span class="p">.</span><span class="n">count</span><span class="o">></span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="s">@"Need >1 test"</span><span class="p">);</span>
<span class="w"> </span><span class="n">test</span><span class="p">.</span><span class="n">readRasterPixels</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">isExhaustive</span><span class="p">;</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">NSXMLElement</span><span class="w"> </span><span class="o">*</span><span class="n">testFile</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">testFileList</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="p">[</span><span class="n">test</span><span class="w"> </span><span class="n">runFileTestWithFileNode</span><span class="o">:</span><span class="w"> </span><span class="n">testFile</span><span class="w"> </span><span class="n">formatName</span><span class="o">:</span><span class="w"> </span><span class="n">fileFormatName</span><span class="w"> </span><span class="n">asRaster</span><span class="o">:</span><span class="w"> </span><span class="n">isRaster</span><span class="p">];</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}];</span>
</code></pre></div>
<p>The actual code we use to add the selector is thanks mostly to a
<a href="http://stackoverflow.com/questions/6357663/casting-a-block-to-a-void-for-dynamic-class-method-resolution">Stack Overflow Posting</a>
which describes exactly how to do this.</p>
<div class="codehilite"><pre><span></span><code><span class="p">+</span> <span class="p">(</span><span class="kt">BOOL</span><span class="p">)</span><span class="nf">CTUTaddInstanceMethodWithSelectorName:</span><span class="p">(</span><span class="bp">NSString</span><span class="w"> </span><span class="o">*</span><span class="p">)</span><span class="nv">selectorName</span><span class="w"> </span><span class="nf">block:</span><span class="p">(</span><span class="kt">void</span><span class="p">(</span><span class="o">^</span><span class="p">)(</span><span class="kt">id</span><span class="p">))</span><span class="nv">block</span>
<span class="p">{</span>
<span class="w"> </span><span class="c1">// don't accept nil name</span>
<span class="w"> </span><span class="n">NSParameterAssert</span><span class="p">(</span><span class="n">selectorName</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// don't accept NULL block</span>
<span class="w"> </span><span class="n">NSParameterAssert</span><span class="p">(</span><span class="n">block</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// See http://stackoverflow.com/questions/6357663/casting-a-block-to-a-void-for-dynamic-class-method-resolution</span>
<span class="w"> </span><span class="kt">id</span><span class="w"> </span><span class="n">impBlockForIMP</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">__bridge</span><span class="w"> </span><span class="kt">id</span><span class="p">)(</span><span class="k">__bridge</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="o">*</span><span class="p">)(</span><span class="n">block</span><span class="p">);</span>
<span class="w"> </span><span class="kt">IMP</span><span class="w"> </span><span class="n">myIMP</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">imp_implementationWithBlock</span><span class="p">(</span><span class="n">impBlockForIMP</span><span class="p">);</span>
<span class="w"> </span><span class="kt">SEL</span><span class="w"> </span><span class="n">selector</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NSSelectorFromString</span><span class="p">(</span><span class="n">selectorName</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">class_addMethod</span><span class="p">(</span><span class="nb">self</span><span class="p">,</span><span class="w"> </span><span class="n">selector</span><span class="p">,</span><span class="w"> </span><span class="n">myIMP</span><span class="p">,</span><span class="w"> </span><span class="s">"v@:"</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
Fastlane + Jenkins Pipelines (Gaige gets his Java on)2019-01-02T22:11:00-05:002019-01-02T22:11:00-05:00Gaige B. Paulsentag:www.gaige.net,2019-01-02:/fastlane-jenkins-pipelines-gaige-gets-his-java-on.html<h2>Jenkins</h2>
<p>For years, I've been using <a href="https://jenkins-ci.org">Jenkins</a> as a CI environment
at ClueTrust. For those unfamiliar with Jenkins, it's a long-running open-source
project built in Java for doing Continuous Integration. It'll work on just about
any platform that can run Java (although it's most at home on Unix machines) and …</p><h2>Jenkins</h2>
<p>For years, I've been using <a href="https://jenkins-ci.org">Jenkins</a> as a CI environment
at ClueTrust. For those unfamiliar with Jenkins, it's a long-running open-source
project built in Java for doing Continuous Integration. It'll work on just about
any platform that can run Java (although it's most at home on Unix machines) and
it can be used with nearly any development toolset. It's been a helpful tool,
but not without it's difficulties.</p>
<p>Most of my personal development for the past few years has been for macOS and
iOS and thus requires Xcode, which means that my Jenkins slaves need to run on
macOS machines. In my case, I run both the slaves and the master on Mac Minis
(including a brand-spanking new one that really screams). Over the years, running
CI on a Mac has been difficult, and required quite a bit of manual tweaking of
the environment to keep up with changes in Apple's toolchain.</p>
<h2>Enter Fastlane</h2>
<p>Over the past few years, development has progressed on an iOS-mostly (macOS-kinda)
development tool called <a href="https://fastlane.tools">Fastlane</a>. The original developer,
<a href="https://twitter.com/KrauseFx">Felix Krause</a> has been working on Fastlane for years
and has moved from independent to Twitter and now Google.</p>
<p>In a nutshell, it shepherds most of the build/test/deploy cycle for iOS and macOS
software, providing pretty output and a fast cycle.</p>
<p>When I went through our last feature release for CartoMobile in the fall of 2018, I
ran into some annoying changes in the build process that caused some difficulties
with building and testing in my desktop and laptop environments,
and especially automating the screenshot process.</p>
<p>I looked to Fastlane as a solution to this problem, and was quite
pleased with the results. In fact, they were so good, I moved my other active
projects, including Cartographica, to Fastlane for build/test/deploy. Despite
comments about Fastlane being mostly for mobile development, I found it quite
easy to work with for macOS development as well.</p>
<p>Of course, if it's worth doing in the manual build process, then it's worth
automating, so I set out to follow the Fastlane instructions to get it working
with Jenkins. It's clear that somebody had the same idea, as there's explicit
support for running Fastlane under Jenkins. Although it required using the
KPP Management Plugin, which I hadn't previously used, it worked quite well.</p>
<h2>Moving to Pipelines</h2>
<p>Fast forward to this winter, and I've been doing some more complex build and
test cycles, and reading up on the more recent techniques for using Jenkins and
decided it was time to try Pipelines. The value proposition was that you could
create a more intelligent pipeline of actions. Whereas Jenkins has long had
the ability to chain builds together, or to use build steps, the proliferation of
individual "projects" necessary to handle a complex build and test cycle could
get pretty unwieldy. For example, Cartographica had a 4-project setup if you
don't count the individual libraries that are built separately.</p>
<p>Applying pipelines to Cartographica wasn't a difficult process, thanks to the
<a href="https://wiki.jenkins.io/display/JENKINS/Convert+To+Pipeline+Plugin">Convert to Pipeline Plugin</a>
provided by Infostretch. The automated conversion worked well, except that it didn't
know how to convert the KPP Management Plugin. After a cursory stop in the Pipeline
Syntax page, it became clear that the KPP Management Plugin was old enough (6 years
since the last update at that point) that it didn't use the required code base to
get automatic support for Pipelines. Sad trombone... But, hey, I'm a programmer,
I've got this.</p>
<h2>Enter the Javas</h2>
<p>I'd done some recent Java work at Haste, but it'd been quite a while since my
last serious effort to pick up someone else's code and run with it in that
language, especially a plugin to yet another, much larger codebase. It turns out
that the Jenkins folks (and the folks who made the original plugin) did a good
job of making things pretty sane. Once I'd reoriented my brain to Java, it was
pretty straightforward to modify the code to support pipelines, and the results
is that after a few hours, I now have a completely functional Jenkins Pipeline
for Cartographica using Fastlane and my modified KPP Management plugin.</p>
<p>My modified source code can be found at <a href="https://github.com/gaige/kpp-management-plugin">gaige/kpp-management-plugin</a>
on <a href="https://github.com">GitHub</a>.</p>
<p>Helpful links:</p>
<ul>
<li><a href="https://jenkins.io/blog/2016/05/25/update-plugin-for-pipeline/">Refactoring a Jenkins plugin for compatibility with Pipeline jobs </a></li>
<li><a href="https://wiki.jenkins.io/display/JENKINS/Plugin+tutorial">Jenkins Plugin Tutorial</a></li>
</ul>
Follow-up on static pages2018-12-16T10:57:00-05:002018-12-16T10:57:00-05:00Gaige B. Paulsentag:www.gaige.net,2018-12-16:/follow-up-on-static-pages.html<p>At the beginning of the month, I <a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">wrote</a>
about the move to convert Gaige's Pages to a static generation model.
Today I'm following up with some performance graphs. There's
absolutely nothing surprising here, but it's good to see nonetheless that things work
as they should.</p>
<a href="https://www.gaige.net/images/large/2018-12-16-Performance-gaigespages.png">
<img src="https://www.gaige.net/images/large/2018-12-16-Performance-gaigespages.png" width=619 height=212>
</a>
<p>Look to the right of …</p><p>At the beginning of the month, I <a href="https://www.gaige.net/gaiges-pages-moves-to-static-generation.html">wrote</a>
about the move to convert Gaige's Pages to a static generation model.
Today I'm following up with some performance graphs. There's
absolutely nothing surprising here, but it's good to see nonetheless that things work
as they should.</p>
<a href="https://www.gaige.net/images/large/2018-12-16-Performance-gaigespages.png">
<img src="https://www.gaige.net/images/large/2018-12-16-Performance-gaigespages.png" width=619 height=212>
</a>
<p>Look to the right of the red line in the middle to see the new site and the left for the
old dynamic site.</p>
<p>As any good programmer does, I am now trading my time to create the posts (x1 for just me)
to reduce the time for readers (x2 at least, one might hope). Even if no human reads
it, I'm saving the indexing bots time.</p>
<p>Average total page load time (DNS+Connection+SSL+First Byte+Download) went from 844ms on
the old site (minimum of 208 and max of 60,169) to 125ms (minimum of 18ms, maximum of 805ms),
so as you can see, our worst-case scenario in the last week was better than the average
in preceding weeks.</p>
<p>As a note for those who are interested, the only significant variation in the previous
dynamic site was the time to first byte. This is the time from after the SSL negotiation
and the pass-off to the CMS and the receipt of the first byte by the browser. This time
has gone from a minimum of 205ms to a maximum of 4ms.</p>
Gaige's Pages moves to static generation2018-11-30T16:30:00-05:002018-11-30T16:30:00-05:00Gaige B. Paulsentag:www.gaige.net,2018-11-30:/gaiges-pages-moves-to-static-generation.html<p>Gaige's Pages has been through a lot of changes over the last 15 years, since I
did the first major revamp of the site. At that time, I was converting from
a statically generated site that I was manually creating (with a little help
from DreamWeaver) to Geeklog, a venerable …</p><p>Gaige's Pages has been through a lot of changes over the last 15 years, since I
did the first major revamp of the site. At that time, I was converting from
a statically generated site that I was manually creating (with a little help
from DreamWeaver) to Geeklog, a venerable, geeky CMS with many more features
than I needed.</p>
<h1>A little history</h1>
<p>At the time (January 2003), I was expecting to create more content; and, boy did
I! During that year, my most prolific blogging year, I managed to crank out
over 1400 individual blog posts, ranging from a 3059-word missive about the
future of media to about 800 pieces less than 100 words in length that mostly
point at other people's thoughts (with some commentary). I wrote about 50 pieces
that would be considered essay-length, which isn't bad, except when you consider
that I really didn't have a full time job that year.</p>
<p>My interest in blogging waned as I got back into working with other people, and
as Carol & I got married in 2004.</p>
<table>
<thead>
<tr>
<th>Year</th>
<th style="text-align:right">Posts</th>
<th style="text-align:right">Word Count</th>
</tr>
</thead>
<tbody>
<tr>
<td>2003</td>
<td style="text-align:right">1493</td>
<td style="text-align:right">214283</td>
</tr>
<tr>
<td>2004</td>
<td style="text-align:right">304</td>
<td style="text-align:right">48612</td>
</tr>
<tr>
<td>2005</td>
<td style="text-align:right">191</td>
<td style="text-align:right">35872</td>
</tr>
<tr>
<td>2006</td>
<td style="text-align:right">149</td>
<td style="text-align:right">30039</td>
</tr>
<tr>
<td>2007</td>
<td style="text-align:right">52</td>
<td style="text-align:right">12761</td>
</tr>
<tr>
<td>2008</td>
<td style="text-align:right">41</td>
<td style="text-align:right">15988</td>
</tr>
<tr>
<td>2009</td>
<td style="text-align:right">15</td>
<td style="text-align:right">7539</td>
</tr>
<tr>
<td>2010</td>
<td style="text-align:right">4</td>
<td style="text-align:right">1418</td>
</tr>
<tr>
<td>2011</td>
<td style="text-align:right">1</td>
<td style="text-align:right">582</td>
</tr>
<tr>
<td>2012</td>
<td style="text-align:right">0</td>
<td style="text-align:right">0</td>
</tr>
<tr>
<td>2013</td>
<td style="text-align:right">5</td>
<td style="text-align:right">1632</td>
</tr>
<tr>
<td>2014</td>
<td style="text-align:right">2</td>
<td style="text-align:right">178</td>
</tr>
<tr>
<td>2015</td>
<td style="text-align:right">7</td>
<td style="text-align:right">2202</td>
</tr>
<tr>
<td>2016</td>
<td style="text-align:right">0</td>
<td style="text-align:right">0</td>
</tr>
<tr>
<td>2017</td>
<td style="text-align:right">1</td>
<td style="text-align:right">574</td>
</tr>
<tr>
<td>2018</td>
<td style="text-align:right">2</td>
<td style="text-align:right">959</td>
</tr>
</tbody>
</table>
<h1>Blog Engines</h1>
<p>Over the intervening 15 years, I've changed blog engines infrequently. Frankly,
it's an annoying process with few significant upsides when you aren't writing
frequently.</p>
<h2>Geeklog (2003)</h2>
<p>I moved to GeekLog in 2003 when I was thinking I wanted to write more and was
tired of dealing with hand-coding HTML (even with the help of DreamWeaver).
This was a reasonable choice at the time, and got me a hybrid HTML editor
and a content indexing system (along with a ton of stuff I didn't need, like
a calendar system).</p>
<p>In 2006, when I acquired Cartographica.com in advance of releasing Cartographica
(in 2008 for beta, and 2009 for public release), I used GeekLog as the basis for
the Cartographica blog as well. This provided me with a converged platform and
the ability to leverage my knowledge.</p>
<h2>SquareSpace (2011+ Cartographica)</h2>
<p>In 2011, as Cartographica was gaining steam, we hired a bright youngster to do
some significant blogging, however GeekLog was showing its age and wasn't really
adequate for blogging with images unless you were a pro with HTML, so I moved
Cartographica to the hosted service <a href="https://www.squarespace.com">SquareSpace</a>,
which we still use.</p>
<h2>Drupal (2013?)</h2>
<p>After moving Cartographica's blog to SquareSpace, I realized that I didn't
particularly enjoy using GeekLog any longer, and support for it was waning.
Wordpress was the new hotness, but it was way too maintenance heavy (see any
security blog), so I wanted something with the right level of geek, good
maintenance and good code hygiene. After consulting with some friends, I
decided on Drupal. Drupal's been good, but I've just not been blogging that
frequently and the amount of effort necessary to maintain and defend a full-fledged
CMS doesn't seem worth the effort for a single-writer blog that is infrequently
updated. In truth, for the last few years I've spent more time updating my
blog software than writing for my blog.</p>
<h2>Static blog engines</h2>
<p>A few months ago, I started looking at moving the Pages to a static blog engine.
My thinking is that I'm not blogging that much these days, and I'd rather use
my scant blog-related time to actually write than to do administrative tasks on
the server; especially those involving security patches. In addition, I prefer to
write my posts in Markdown. Most of the static blogging engines work with Markdown
and that just seemed the right direction for me to take.</p>
<h3><del>Jekyll</del></h3>
<p>I really tried to use Jekyll. <a href="https://technotes.seastrom.com">RS</a>, my oft-times
partner in technology, considers it "good enough to offset being one of two
unfortunate cases of ruby in my life, the other being brew.sh".
It seems to be well liked by most people who
use it, so I set out to translate my 2000+ blog posts into Markdown for
assimilation into Jekyll.</p>
<p>Unfortunately, I ran into two problems: first, it's good for small blogs, but
regeneration times on large ones is really long; second, it's written in Ruby.
I know what you're thinking: I shouldn't be bashing an entire programming
language which is likely used by millions of people. Well, millions of people
smoke, too, and although both things are completely legal, I wouldn't consider
either safe for your health.</p>
<p>It's not so much the ruby syntax that bothers me, it's the ethos. Jekyll is great
for people who like it and have no problems with it, but when my generation was
taking literally hours to never finish, I decided that I'd take a look at the
code to make sure there wasn't some weird bug I was triggering.</p>
<p>We'll never know if that was the case, because after finding no reasonable way
to even figure out what module was consuming time, I just gave up.</p>
<h3>Pelican</h3>
<p>I took to Google to find something that fit the bill. If I was going to have to
potentially debug this puppy, I wanted it to be written in a language that I am
comfortable with and one whose ethos involves making code that can be read and
debugged by someone other than the original author.</p>
<p>Into google, I typed <code>static blog engine python markdown jinja2</code>. I wasn't sure
I'd find something, but I wanted <code>python</code> for maintainability, <code>static blog engine</code>
so that I didn't have to work if I wasn't changing anything, <code>markdown</code> for a
fast and familiar writing environment (Hello, <a href="https://www.barebones.com">BBEdit!</a>)
and finally <code>jinja2</code> for any templating.
The heavier-duty <a href="https://www.macgis.com">MacGIS</a> web site had already been moved
to Django, which uses <code>Python</code> and (optionally) <code>jinja2</code> as a templating language.
Similarly, I'd moved from <a href="https://puppet.org">Puppet</a> to <a href="https://ansible.org">Ansible</a>
a couple of years ago (another story, many thanks to Rob), which is also based on
<code>Python</code> and <code>jinja2</code>.</p>
<p>The first recommendation was <a href="https://github.com/getpelican/pelican"><code>Pelican</code></a>,
which has been my answer for this stage of Gaige's Pages.</p>
<p>The transition has not been without hiccups, and I even ran into a bug which I
needed to diagnose; but, diagnose it I did, because I could use all of my tricks
for debugging <code>Python</code>, including <a href="https://jetbrains.com/pycharm">PyCharm</a></p>
Looking for a Nav system (Revisited: 2018)2018-11-30T15:00:00-05:002018-11-30T15:00:00-05:00Gaige B. Paulsentag:www.gaige.net,2018-11-30:/looking-for-a-nav-system-revisited-2018.html<p>If you want a good feel for the advancements in Navigation systems in the last
10 years, you should check out my piece
<a href="https://www.gaige.net/looking-for-a-nav-system.html">Looking for a Nav system</a>
from 2008. The article went through my key issues that lead to my recommendation
of the TomTom's in those days.</p>
<p><strong>TL;DR …</strong></p><p>If you want a good feel for the advancements in Navigation systems in the last
10 years, you should check out my piece
<a href="https://www.gaige.net/looking-for-a-nav-system.html">Looking for a Nav system</a>
from 2008. The article went through my key issues that lead to my recommendation
of the TomTom's in those days.</p>
<p><strong>TL;DR:</strong> Today, I just think you should use whatever is best integrated with your phone
or what gives you the best traffic or routes in your most frequent driving
areas.</p>
<p>Running through the items from the article:</p>
<ol>
<li>
<p><strong>The Nav system should (mostly) trust you</strong>
In the original article, I stated "believe" you, but I think trust is the real
issue. If I drive in a different direction, the unit should figure there may
have been a good reason for it and give some priority (at least a tie breaker)
to the human driver.</p>
</li>
<li>
<p><strong>MapShare updating technology</strong>
I thought this was going to be a helpful tech at the time, and it is, but it's
now supplanted by much better connected algorithms for navigation. Not only
does your Nav system believe you, but if you drive over a road that the
device doesn't know about often enough, there's a good chance it'll show up
in a future update.</p>
</li>
<li>
<p><strong>Better pricing on map updates</strong>
Not much better pricing than free, or at least included with your device. Take
your pick of how to pay: subscription for "pro" software, free for Apple's
defaults, or with your data in the case of everyone else. But, you generally
won't be paying a fee unless you are using pro software.</p>
</li>
<li>
<p><strong>Better Management Software</strong>
No need for this any longer. Once your mapping system locates you, as long
as you're online, you have the maps you need. There are some systems designed
to allow you to choose the areas to cache, but generally that's not necessary
except for specialized systems for activities like hiking.</p>
</li>
<li>
<p><strong>IQ Routing</strong>
Thanks to the connected nature, this is a non-issue. Everyone has segment-based
timing these days, automatically reported back; usually while you drive.</p>
</li>
<li>
<p><strong>Advanced Lane Guidance</strong>
Table stakes again.</p>
</li>
</ol>
<p>There are still some difficulties:</p>
<ul>
<li><strong>mounting brackets are still a problem</strong>
Unless you have a navigation system that coordinates with your car
(Apple's CarPlay and Google's equivalent, Android Auto), you're probably
struggling with some difficult to handle device to hold your phone.</li>
<li><strong>Privacy</strong>
Now that everyone's driving is being monitored by their navigation system,
some amount of private information that was previously unavailable is now
available to bad actors. You may trust your phone provider and OS manufacturer
(although that may be a bad idea in some cases), you have to watch any
App that has access to your location outside of when the App is running.</li>
<li><strong>Cell coverage == navigability</strong>
It's getting better, but in many cases, if you lose your cell coverage, your
nav system not only loses its ability to get real-time traffic data, but you
may be stranded without a map.</li>
</ul>
<p>Tune in in 2028, and maybe I'll update this topic again!</p>
Codesigning ate my Sunday2018-11-11T16:49:00-05:002018-11-11T16:49:00-05:00Gaige B. Paulsentag:www.gaige.net,2018-11-11:/codesigning-ate-my-sunday.html<p>I have a version of Cartographica that I need to push out before the end of the
year, due to a certificate expiration on one of my long-term servers. As a
bulwark against problems occurring just at the turn of the year and to make sure
that users can use …</p><p>I have a version of Cartographica that I need to push out before the end of the
year, due to a certificate expiration on one of my long-term servers. As a
bulwark against problems occurring just at the turn of the year and to make sure
that users can use the 1.4.x series of Cartographica, I set out to sign and
release 1.4.9, a version of 1.4.8 containing only this signature fix.</p>
<p>And then, codesigning ate my Sunday.</p>
<p>It seemed simple enough, I ran my release build script, it created a disk image
(dmg) file and pushed it up to our web server for release. Then I added the
release notes into Feeder.app (which I use to push the RSS feeds) and ran one
last test before I pushed it out to the users:</p>
<ul>
<li>File downloaded fine</li>
<li>Disk Image opened fine</li>
<li>Cartographica application copied fine</li>
</ul>
<p>And then Cartographica crashed upon start, giving the relatively cryptic:</p>
<pre> Reason: no suitable image found. Did find:
/Applications/Cartographica.app/.../XXX.dylib: code signing blocked mmap() of /Applications/Cartographica.app/.../XXX.dylib
</pre>
<p>Well, that was unexpected. What ensued was a wide variety of searches on the
internet mostly leading to people's expired keys and certificates or signing and
testing the wrong items, but eventually I noticed that the library in question
(supplied by a third party) was a fat library (i386 an x64_86). Could that be
related?</p>
<p>It appears that version 2 code signatures contain the following "Format=bundle
with Mach-O thin (x86_64)" which is the same as our last shipping version in
August 2017, but the 2017 variant works just fine. Maybe something related to
Mojave (on which I'm running the signing and the executing the binary)?</p>
<p>I lipo'd the library to remove the i386 code and rebuilt. Then I re-signed the
binary. Seems fine on pre-10.14 machines, but won't work on pre-10.14. Even
if I sign on 10.13, it works fine on 10.13, but not on 10.14.</p>
<p>I spent time looking through any number of internet postings about similar
problems (mostly iOS, which wasn't necessarily applicable) and they were mostly
related to the actual certificates being used to sign. You may want to give
those a try if you got here looking for a solution to this problem on iOS...</p>
<p>After about 5 hours I finally decided to throw in the towel, save for one last
effort. I rebuilt with code signing on my MacBook Pro (still running 10.13,
because I hadn't had time to upgrade it yet) and Xcode 7 (the last version that
can build the 1.4 string without substantial updates. Under normal
circumstances, I don't build the binaries with code signing on (at least not in
the older versions) because of inconsistencies in the way that code signing
worked. Further, it was just as easy to re-sign everything as part of my
scripted steps to get the DMG file built and uploaded.</p>
<p>This time, I changed the code signing parameters to sign with my Developer ID
certificate and rebuilt for archive, taking that archive and manually deploying
the disk image (thanks to <a href="https://c-command.com/dropdmg/">DropDMG</a>, this was a
simple command line operation). The build was successful and ran on 10.13
without a hitch.</p>
<p>Testing on 10.14 also ran without a hitch, so it looks like I've finally solved
the last 1.4 issue.</p>
TPS Reports? (Testing PostgreSQL under SmartOS)2018-10-22T06:47:00-04:002018-10-22T06:47:00-04:00Gaige B. Paulsentag:www.gaige.net,2018-10-22:/tps-reports-testing-postgresql-under-smartos.html<p>Rob and I are working on updating our standard environmnent in our data
centers. As may be clear already, we're big proponents of SmartOS,
which has been working really well for our needs. We've also big
proponents of automation (and, in particular, Ansible).</p>
<p>Due to a hardware failure this weekend …</p><p>Rob and I are working on updating our standard environmnent in our data
centers. As may be clear already, we're big proponents of SmartOS,
which has been working really well for our needs. We've also big
proponents of automation (and, in particular, Ansible).</p>
<p>Due to a hardware failure this weekend, we had to promote our test
environment to a production machine while waiting for a power supply to
be sourced (yeah, our next environments are going to be dual power
supply, now that we have dual feeds to the rack). This left me without
a test box, which turned out to be more of a problem than I'd realized,
because some of my automated tests for my macOS and iOS development
require a functioning server.</p>
<p>Enter Toys-R-Us... well, of course, Toys-R-Us has exited, but in doing
so, they provided us the opportunity to pick up some recent-model Dell
kit at ridiculously low prices. As Rob had recently installed my two
machines (R330s) in the rack, I now have the opportunity to take them
for a real-world spin. As an added bonus, we've got 10K 300GB SAS
drives in one (running mirrored) and 500GB SATA SSDs (Samsung 850 EVO)
running in the other (also mirrored). Compare and contrast this to our
existing HP DL160g6 servers which have RAIDZ2 4-drive x 4TB 5400RPM WD
Red drives, and it looks like we've got an opportunity for a bit of a
bake off.</p>
<p>My first test is to take a look at the PostgreSQL performance on these
systems. Apples-to-Apples will be a bit harder, but we've got some
interesting comparisons here anyway, so I've rolled out <code>pgbench</code> for
the testing and have chosen to simulate my workload using
<code>TPC-B (sortof)</code>, which is the tool's default with 100 clients on 10 threads, with
100 transactions per (<code>pgbench -c 100 -j 10 -t 100</code>).</p>
<table>
<thead>
<tr>
<th>System</th>
<th style="text-align:right">Latency Average</th>
<th style="text-align:right">TPS (w/estab)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Dell R330 w/SSD</td>
<td style="text-align:right">135.242</td>
<td style="text-align:right">739.415489</td>
</tr>
<tr>
<td>Dell R330 w/Quickly Spinning Rust</td>
<td style="text-align:right">456.189</td>
<td style="text-align:right">219.207000</td>
</tr>
<tr>
<td>HP DL160g6 w/Spinning Rust</td>
<td style="text-align:right">996.330</td>
<td style="text-align:right">100.368323</td>
</tr>
</tbody>
</table>
<p>So, there you have it, the not-so-surprising news that mirrored SSDs are
faster than mirrored 300GB 10K SAS drives when running certain
PostgreSQL benchmarks on a system doing nothing else.</p>
Jumping a dead 2000 Boxster S2017-10-21T12:49:00-04:002017-10-21T12:49:00-04:00Gaige B. Paulsentag:www.gaige.net,2017-10-21:/jumping-a-dead-2000-boxster-s.html<p>I'm generally extremely happy with my 17-year old 2000 Boxster S that I bought
new. However, running back and forth between Atlanta means that I've not been
driving it as much as I have recently (although quite a bit more than I did in
the 2008-2010 period). Last year, I …</p><p>I'm generally extremely happy with my 17-year old 2000 Boxster S that I bought
new. However, running back and forth between Atlanta means that I've not been
driving it as much as I have recently (although quite a bit more than I did in
the 2008-2010 period). Last year, I replaced the battery with the same basic
unit with a 48-month warranty, figuring that 7 years was a long time for a
battery to last (yes, the previous one was replaced in 2010).</p>
<p>After returning from one of my trips back home, I tried to start the car,
resulting in a very disappointing lack of lights when I opened the vehicle. I
figured it was probably time to appropriate one of those nifty battery-powered
jumps starting units (which I did, more on that later).</p>
<p>However, when I approached the car, I realized that getting the hood open (or
front trunk, or bonnet, depending on your opinion on such things). The 2000
model has manual trunk releases, but the central locking system protects those
releases through the use of a solenoid-actuated locking bar. Without power, that
bar doesn't move.</p>
<p>After scouring the internet, I found a lot of information about getting past
this problem on different models, but each of them is a bit different. Models a
couple of years younger (I believe 2003+) have a terminal in the fuse box to
energize the battery remotely to get past this. However, my model requires a
slightly hackier operation.</p>
<p>Required items for this operation:</p>
<ul>
<li>12V supply sufficient to start the car (I used a battery jump unit, but another car will also do)</li>
<li>6"+ piece of wire stripped at both ends</li>
<li>Jumper cables</li>
<li>(Your keys)</li>
</ul>
<h2>Operation</h2>
<ol>
<li>Open the door using the key to access the interior of the vehicle</li>
<li>Remove the fuse box cover (located in the driver's side footwell)</li>
<li>Using the fuse-removal tool, remove the C3 fuse (you should check the
enclosed fuse map and look for "Central Locking", but it's C3 on my vehicle and
elsewhere on the internet)</li>
<li>Insert a piece of wire in the C3 socket and replace the fuse</li>
<li>Connect the negative end of your 12V supply to the body of the car (I used
the door hinge)</li>
<li>Connect the positive end of your 12V supply to the other end of the wire
connected to the C3 fuse socket
At this point, you should see interior lights energize and your fan may come
on. If you don't see signs of life, there's something wrong with the wire
placement or the fuse (or a deeper problem with your vehicle)</li>
<li>Rotate the door latch manually (to convince the car that the driver's side
door is shut)
This is essential and cost me a couple of weeks (time, not effort) in trying
to figure this out</li>
<li>Now use the key in the driver's side door to lock and unlock the doors
You should hear the click as the solenoid pulls the locking bar back and you
should be able to freely operate the hood latch</li>
<li><strong>Pull the door handle to unset the door latch - Failure to do this could
damage your door system</strong></li>
<li>Detach your cables from the car and remove the wire</li>
<li>Now, pop the hood and complete the jump-start operation</li>
</ol>
Sun SparcStation 20 vs Raspberry Pi2015-12-27T12:17:00-05:002015-12-27T12:17:00-05:00Gaige B. Paulsentag:www.gaige.net,2015-12-27:/sun-sparcstation-20-vs-raspberry-pi.html<p>For those of us old timers, here's an amusing <a href="http://eschatologist.net/blog/?p=266">shootout between a SparcStation
20 and a Raspberry Pi</a>, including both
the Pi and the Pi2 (but unfortunately not the Pi0).</p>
<p>For those unfamiliar with the SS20, it was a workhorse desktop in the 1990's.
Early pizza-box design and pricing out …</p><p>For those of us old timers, here's an amusing <a href="http://eschatologist.net/blog/?p=266">shootout between a SparcStation
20 and a Raspberry Pi</a>, including both
the Pi and the Pi2 (but unfortunately not the Pi0).</p>
<p>For those unfamiliar with the SS20, it was a workhorse desktop in the 1990's.
Early pizza-box design and pricing out at well over $10,000 at the time
(according to the article, closer to $25K today). It was an awesome desktop
workstation, and was seen in some equipment racks as a "cheap" alternative
server platform. For comparison, the Pi sells for $20, and the Pi2 for $35.</p>
<p>The short version is that a computer which costs roughly 1/1000 of the price
performs about 7x as fast for operations (and for the record, there is no
category where the SparcStation wins). For those keeping score, that's a 7000x
increase in price/performance in 20 years.</p>
SmartOS, Postfix and IPv62015-12-18T06:46:00-05:002015-12-18T06:46:00-05:00Gaige B. Paulsentag:www.gaige.net,2015-12-18:/smartos-postfix-and-ipv6.html<p>As part of completing our shut-down of 2007-vintage Xserves at the hosting
center, we're moving a lot of servers to SmartOS (or at least SmartOS-hosted
VMs). We've been really happy with the system so far. Here's a quick story of
the power of this environment.</p>
<p>As part of the transition …</p><p>As part of completing our shut-down of 2007-vintage Xserves at the hosting
center, we're moving a lot of servers to SmartOS (or at least SmartOS-hosted
VMs). We've been really happy with the system so far. Here's a quick story of
the power of this environment.</p>
<p>As part of the transition, I decided to take care of a long-standing intention
to allow IPv6 mail delivery. The latest releases make IPv6 much easier to deal
with in SmartOS and so I figured I'd turn it on and see how it went. Early
tests looked good, but there was little IPv6 traffic initially. However, once
google/gmail started throwing data our way, it became clear there was a
problem. IPv6 reverse lookups were failing in Postfix and that was causing
mail not to be delivered. Thankfully, it was a 450 error, so the mail would
get queued on the server and be tried again later, but it wasn't a good place
to be, so we turned off IPv6 and went in search of the root cause.</p>
<p>A quick look at the postfix source code made it clear that the OS (and not
postfix's internal DNS code) was being used to look this up. I decided to
play with it a little longer, grab source code so that I could do some
experiments, and then I went to bed.</p>
<p>In the morning, I did what you do when you have standard configurations and
SmartOS: I stood up a new server and ran some tests on it. The new server was
configured with the latest (15.3.0) build of SmartOS and the 2015Q3 packages,
resulting in an upgrade to the 3.0.2 version of Postfix. Once set up, I ran
some email through it and was able to confirm whether this resolved the
problem, and it did.</p>
<p>The moral of the story (which I wish I'd realized a few hours earlier
yesterday): here in the future, it's easier to stand up another machine to
check whether a bug has been fixed than to read the release notes and source
code.</p>
<p>From a discovery/knowledge perspective, I'm not sure I like this new world.
From an efficiency perspective, disposable virtual hardware is a luxury and a
huge time saver.</p>
Replacing a RAID set under El Capitan2015-12-04T07:32:00-05:002015-12-04T07:32:00-05:00Gaige B. Paulsentag:www.gaige.net,2015-12-04:/replacing-a-raid-set-under-el-capitan.html<p>Over Thanksgiving, one of the two drives in my "Big Disk" RAID (it was a
mirror of 2 2TB drives that I used to store large things that aren't worth
having on the SSD on my Mac Pro). Generally speaking, my response to
failures with SMART (especially with cheap spinning …</p><p>Over Thanksgiving, one of the two drives in my "Big Disk" RAID (it was a
mirror of 2 2TB drives that I used to store large things that aren't worth
having on the SSD on my Mac Pro). Generally speaking, my response to
failures with SMART (especially with cheap spinning rust drives) is to replace
the drive immediately and if it's a set of drives in a RAID to consider
replacing both of them and bumping to the next most efficient capacity.</p>
<p>For this year's upgrade (it's actually been 2+ years since the time these
drives were installed), the most cost efficient drives are 4TB. I don't think
I'm likely to fill them up any time soon, but pictures and video aren't
getting smaller, and I sometimes have to store some large GIS database on
these while I'm working with them, and the new drives are NAS rated and have a
3 year warranty, so I need to anticipate use for 3 years out.</p>
<p>With El Capitan, Apple's Disk Utility program has become... well, pretty
lackluster. However, from a RAID perspective, I've used the command line for
years because of the desire to have explicit control over what's being done.</p>
<p>The key objectives:</p>
<ul>
<li>Replace the 2TB mirrored RAID with a 4TB mirrored RAID</li>
<li>Retain the existing Time Machine history</li>
<li>Retain the existing CrashPlan history</li>
</ul>
<p>The process was quick and successful, and here are the steps that I took:</p>
<ol>
<li>
<p>Using an external SATA to Thunderbolt dock, I attached and formatted the new 4TB drive, naming it something other than it's final name. Because of the way OS X handles name collisions, I didn't want any confusion in the devices, so I chose to only have one drive mounted with a particular name at a particular time.</p>
</li>
<li>
<p>Using <a href="https://bombich.com">Carbon Copy Cloner</a>, I cloned the old 2TB RAID's contents onto the new 4TB drive. Averaging >100MB/s, this still took a while (about 4 hours for the 1.4TB of active storage)</p>
</li>
<li>
<p>Once the clone was complete, I unmounted the 2TB RAID and pulled both of the constituent drives from the <a href="http://www.lacie.com/products/thunderbolt/5big-thunderbolt-2/">LaCie 5Big Thunderbolt</a> (link is to the TB2 version of this device) and set them aside.</p>
</li>
<li>
<p>Before setting the old drives down, I marked them with a Sharpie, taking care to mark the <strong>FAILED</strong> drive so that it could be destroyed, and the <strong>OK</strong> drive so that it could be retained if desired as a snapshot. I'll keep it for at least a few weeks, but then it'll either go to the safe deposit box, or the disk shredder. Drives are always marked with the RAID name and date they were taken out of service, along with any appropriate legends (Personal and Confidential, copyright, etc)</p>
</li>
<li>
<p>Unmount the 4TB drive from the external SATA to Thunderbolt dock and install it, and it's future mirror into the enclosure</p>
</li>
<li>
<p>Once the 4TB drive remounts from the enclosure, make sure that Time Machine isn't running (and either wait for it to stop, or cancel the run before proceeding)</p>
</li>
<li>
<p>Now, convert the 4TB drive to the new RAID by using Terminal and the command line:
diskutil appleRAID enable mirror /dev/disk3s2
You will need to replace <em>/dev/disk3s2</em> with the slice number corresponding to
the volume on the disk, not the disk device itself. This can be gathered by
using diskutil's list command if you are uncertain.</p>
</li>
<li>
<p>Rename the 4TB drive to the name of the original RAID (this takes care of the CrashPlan issue, since CrashPlan only knows the path to volumes)</p>
</li>
<li>
<p>Now that we have the RAID set prepared and the volume renamed, we need to make sure that Time Machine thinks it's the same drive for content purposes. To do this we'll use Terminal again:
sudo tmutil associatedisk -a "/Volumes/RAID"
/Volumes/TM/Backups.backupdb/Machine/Latest/RAID
Names have been changed for privacy, but the basics are clear, you need to
replace <em>/Volumes/RAID</em>** **with the UNIX path of your volume as mounted on
your computer and the second path is the path to the volume name in the
<em>Latest</em> Time Machine backup. Do not associate with another backup or you'll
get an error.</p>
</li>
<li>
<p>To confirm this process, you can start a TM backup and see if you see a message from backupd in the <strong>Console</strong> app like this:
com.apple.backupd[1931]: Inheritance scan may be required for '/Volumes/RAID',
associated with previous UUID: XXXXX-XXXX-XXXX-XXXX-XXXXXXX
Indicating that Time Machine will try and sync up the volume histories.</p>
</li>
<li>
<p>Once the initial Time Machine backup is complete, start rebuilding the RAID (this is going to take a <em>long</em> time, which is why I do it after the initial TM backup), using <strong>Terminal</strong> again:
diskutil appleRAID add member /dev/disk4 disk7
replacing <em>/dev/disk4</em> with the device you want to add, and <em>disk7</em> with the
RAID's Device Node (you can again find these using diskutil's list command)</p>
</li>
</ol>
<p>A bit time consuming, but everything worked like a charm, and I have full
access to my Time Machine history for my new RAID set.</p>
The Age of Deception2015-10-22T06:20:00-04:002015-10-22T06:20:00-04:00Gaige B. Paulsentag:www.gaige.net,2015-10-22:/the-age-of-deception.html<p>Occasionally, in the vast expanse of the internet there are gems from people I
know and respect. I'm not going to summarize, because the entire article,
<a href="http://stephenhultquist.com/thoughts/2015/09/24/the-age-of-deception/">The Age of Deception</a> , is worth reading by itself.</p>
<p>Thanks, ssh.</p>
Obama Won't Seek Access to Encrypted User Data2015-10-22T05:07:00-04:002015-10-22T05:07:00-04:00Gaige B. Paulsentag:www.gaige.net,2015-10-22:/obama-wont-seek-access-to-encrypted-user-data.html<p>Somehow in the midst of all of the craziness around here, I missed that, as
the New York Times reports, <a href="http://www.nytimes.com/2015/10/11/us/politics/obama-wont-seek-access-to-encrypted-user-data.html?smprod=nytcore-iphone&smid=nytcore-iphone-share&_r=0">Obama Won’t Seek Access to Encrypted User
Data</a>. For the time being, they appear to have agreed to the rationale
that a back door provides as much entree to …</p><p>Somehow in the midst of all of the craziness around here, I missed that, as
the New York Times reports, <a href="http://www.nytimes.com/2015/10/11/us/politics/obama-wont-seek-access-to-encrypted-user-data.html?smprod=nytcore-iphone&smid=nytcore-iphone-share&_r=0">Obama Won’t Seek Access to Encrypted User
Data</a>. For the time being, they appear to have agreed to the rationale
that a back door provides as much entree to the criminal element as it does to
law enforcement, and that the benefits don't exceed the costs. I'm sure that
part of this is also due to the political capital that would have to be
expended to go against the interests of the tech industry, and the potential
economic damage if the US continues to be a place where it is believed that
the government has ready access to stored and in-flight data.</p>
<p>At least for the time being, this is good news for US companies that want to
compete in international markets (especially Europe) where uer data protection
is given more weight (in law, at least) than it is here.</p>
<p>The <a href="https://eff.org">EFF</a> has a <a href="https://www.eff.org/deeplinks/2015/10/partial-victory-obama-encryption-policy-reject-laws-mandating-backdoors-leaves">more cautionary take</a> on this announcement.</p>
Familial DNA Searching2015-10-22T04:59:00-04:002015-10-22T04:59:00-04:00Gaige B. Paulsentag:www.gaige.net,2015-10-22:/familial-dna-searching.html<p>Wired had an article last week entitled <a href="http://www.wired.com/2015/10/familial-dna-evidence-turns-innocent-people-into-crime-suspects/">Your Relative's DNA Could Turn You
Into a Suspect</a>, in which they describe a of using
familial DNA searching to locate suspects. There are interesting implications
here, especially with regard to public DNA search resources like Ancestry.com.</p>
<p>Thanks to <a href="https://www.schneier.com/blog/">Bruce Schneier's Blog …</a></p><p>Wired had an article last week entitled <a href="http://www.wired.com/2015/10/familial-dna-evidence-turns-innocent-people-into-crime-suspects/">Your Relative's DNA Could Turn You
Into a Suspect</a>, in which they describe a of using
familial DNA searching to locate suspects. There are interesting implications
here, especially with regard to public DNA search resources like Ancestry.com.</p>
<p>Thanks to <a href="https://www.schneier.com/blog/">Bruce Schneier's Blog</a> for the
link.</p>
Time (Saver) Machine2015-08-09T09:14:00-04:002015-08-09T09:14:00-04:00Gaige B. Paulsentag:www.gaige.net,2015-08-09:/time-saver-machine.html<p>Over the past couple of weeks, I once again reacquainted myself with the joy
of using TimeMachine as a backup system. (Please, use more than one, at least
one off-site and one on-site would be a good idea, consider CrashPlan for the
offsite version, we've used it for years and …</p><p>Over the past couple of weeks, I once again reacquainted myself with the joy
of using TimeMachine as a backup system. (Please, use more than one, at least
one off-site and one on-site would be a good idea, consider CrashPlan for the
offsite version, we've used it for years and are very happy with it).</p>
<p>In this case, I needed to borrow a computer from my cluster of Mac Minis that
make up my build and testing farm. In particular, we needed a clean machine
that could be used and wiped, so I wanted to back up the system, install a
fresh copy of OS X 10.10.4 (the machine was running pre-release 10.10.5), take
it to the trade show, and then reverse the process when I came back.</p>
<p>My network environment at home includes a Mac Mini server running an older
version of the OS and serving as a Time Machine server to machines that aren't
easily connected directly to disk. This is an easy configuration to use if
you have any machine that is running all of the time, especially with recent
versions of OS X that can have server grafted on.</p>
<p>The process went without a hitch. I hadn't been backing up that machine
(that was a bit of a surprise, since I thought I'd learned my lesson a couple
of years ago when I lost the hard drive on another one of my Mac Minis), but
turning on Time Machine was easy and the backup (over wired Ethernet) went
quickly, especially since that machine is basically a simple test box, so it
doesn't have much installed on it.</p>
<p>Installation was a bit more of a hassle, since I normally run that machine
without a monitor, I needed to hook it up to my TV to install (have I
mentioned recently how much I enjoy the fact that we have HDMI these days?),
and the USB stick I was installing from wasn't USB 3, so it was a bit slow.</p>
<p>The machine worked great at the conference, and since it didn't have anything
of value on it, I just restored over top of it (after erasing the hard drive
partition) using Recovery Mode. My third-party Bluetooth keyboard didn't help
much when trying to hold CMD-R to bring up Recovery Mode, so I had to drag
down the one remaining USB keyboard in order to boot it. Once that boot was
done, I selected my Time Machine server, logged in, and chose the backup to
restore. Wait for a little while for the recovery to complete and wow!
Completely functioning machine!</p>
<p>If we were doing a lot of shows (especially with dicey environments like Black
Hat), then I would probably have bought a machine just to use for this
purpose, but in this case, the cost was only a couple of minutes of my time
and a couple hours of the device's time. All told, a great experience.</p>
Academia's Tug-of-war with the NSA over Encryption2014-11-18T06:33:00-05:002014-11-18T06:33:00-05:00Gaige B. Paulsentag:www.gaige.net,2014-11-18:/academias-tug-of-war-with-the-nsa-over-encryption.html<p>There's an excellent article, <a href="https://medium.com/stanford-select/keeping-secrets-84a7697bf89f">Keeping Secrets</a>, on <a href="https://medium.com">medium</a> today
(originally from the <a href="https://stanfordmag.org/contents/keeping-secrets">November/December 2014 issue of Stanford
Magazine</a>) about the
conflict between academic work on cryptography and the NSA's role in national
security. Most of the focus is on what happened and not on who was right or
wrong …</p><p>There's an excellent article, <a href="https://medium.com/stanford-select/keeping-secrets-84a7697bf89f">Keeping Secrets</a>, on <a href="https://medium.com">medium</a> today
(originally from the <a href="https://stanfordmag.org/contents/keeping-secrets">November/December 2014 issue of Stanford
Magazine</a>) about the
conflict between academic work on cryptography and the NSA's role in national
security. Most of the focus is on what happened and not on who was right or
wrong. Particularly interested is the early section about the increasing
understanding of the necessity of encryption for data security and computer
communication that started as early as the 1970's. Once again, thanks to
Bruce Schneier's excellent blog, <a href="https://www.schneier.com">Schneier on Security</a>, for the pointer.</p>
Nice retrospective podcast on Real Genius2014-07-14T05:49:00-04:002014-07-14T05:49:00-04:00Gaige B. Paulsentag:www.gaige.net,2014-07-14:/nice-retrospective-podcast-on-real-genius.html<p>If you have interest at all in 1980's "culture", the tech industry, and/or the
movie <a href="http://www.imdb.com/title/tt0089886/combined"><em>Real Genius</em></a>, you
should check out the <a href="http://www.imore.com/reviewcast">iMore Review</a> program
<a href="http://www.imore.com/review-16-real-genius">Review 16: Real Genius</a>.</p>
<p>Don Melton, Matt Drance, Guy English, and Rene Ritchie do a great job of
running down the highs and lows …</p><p>If you have interest at all in 1980's "culture", the tech industry, and/or the
movie <a href="http://www.imdb.com/title/tt0089886/combined"><em>Real Genius</em></a>, you
should check out the <a href="http://www.imore.com/reviewcast">iMore Review</a> program
<a href="http://www.imore.com/review-16-real-genius">Review 16: Real Genius</a>.</p>
<p>Don Melton, Matt Drance, Guy English, and Rene Ritchie do a great job of
running down the highs and lows of this classic.</p>
Nice set of Nagios scripts for OS X2013-12-12T04:05:00-05:002013-12-12T04:05:00-05:00Gaige B. Paulsentag:www.gaige.net,2013-12-12:/nice-set-of-nagios-scripts-for-os-x.html<p>When digging around for information about Apple's new Caching Server, I
happened across this informative article about <a href="http://www.yesdevnull.net/2013/10/os-x-mavericks-server-setting-up-caching-server/">Caching Server for
Mavericks</a> by Dan Barrett. Definitely worth a read if you're
interested in finding out how to make the most of your network connection with
your Macs.</p>
<p>However, from there, I …</p><p>When digging around for information about Apple's new Caching Server, I
happened across this informative article about <a href="http://www.yesdevnull.net/2013/10/os-x-mavericks-server-setting-up-caching-server/">Caching Server for
Mavericks</a> by Dan Barrett. Definitely worth a read if you're
interested in finding out how to make the most of your network connection with
your Macs.</p>
<p>However, from there, I noticed a link to the <a href="https://github.com/jedda/OSX-Monitoring-Tools">Nagios plugins for OS
X</a> on github. The plugins
contain a lot of useful functionality for monitoring OS X systems, and appear
to support many versions, supposedly back to 10.4. They are all local
scripts, so they need to run on the systems that you are checking.</p>
<p>There's a lot of useful stuff here.</p>
Great piece on user interface evolution2013-05-13T07:15:00-04:002013-05-13T07:15:00-04:00Gaige B. Paulsentag:www.gaige.net,2013-05-13:/great-piece-on-user-interface-evolution.html<p>Matt Gemmell penned a great piece on User Interface design and evolution as it
relates to, well, a lot of things. It's definitely worth the time to read
<a href="https://web.archive.org/web/20210412222742/https://mattgemmell.com/tail-wagging/">Tail Wagging</a>.</p>
Backup Software2013-03-31T08:05:00-04:002013-03-31T08:05:00-04:00Gaige B. Paulsentag:www.gaige.net,2013-03-31:/backup-software.html<p>I'd never heard of <a href="http://www.worldbackupday.com">World Backup Day</a> before
seeing an article about it in <a href="http://wired.com">Wired</a> today, but it sounds
like a good idea, especially for those whose friends may partake in a little
bit of the April Foolery tomorrow.</p>
<p>So, it's a good time for me to discuss backup software …</p><p>I'd never heard of <a href="http://www.worldbackupday.com">World Backup Day</a> before
seeing an article about it in <a href="http://wired.com">Wired</a> today, but it sounds
like a good idea, especially for those whose friends may partake in a little
bit of the April Foolery tomorrow.</p>
<p>So, it's a good time for me to discuss backup software and strategies. I'm not
going to speak specifically about how I perform backups, but here are some key
packages and concepts that are good to think of when you are considering a
backup strategy. And, yes, this should be a strategy, not a specific backup.
Unless you feel that all of your data is easily replaceable (your photos, your
business plan, your accounting data, your scanned legal documents, etc), you
need to take this seriously.</p>
<h2>Locations</h2>
<p>Sometimes people ask me where they should store their backups. My belief is
that a minimum of two physically distant locations are a must. Keeping backups
at home is cheap and convenient, and results in complete loss in the case of a
fire. Those of you with "fire safes" need to keep in mind that most "fire
safes" are rated for documents. The general rule is that they keep things
cooler than 350° which is great, except that slides, CDs, DVDs, etc. tend to
become useless after any reasonable amount of time spent at 125° and above, so
if you are going to keep your backup safe in a safe, then make sure you have
something that is media rated, not document rated.</p>
<p>And then think about flooding, earthquakes, and sinkholes. Each of these can
take out your media pretty easily and irreparably, and if your main copy of
your data is also in your house, you're looking at a total loss.</p>
<p>Safe Deposit Boxes are a reasonable place to keep spare hard drives and CD/DVD
copies of data. They're unlikely to fail in the same way as your home or
laptop, so you have some diversity, and generally speaking they are safe. Most
folks don't encrypt data which goes to a safe deposit box, and that's a two-
edged sword: your data is in its easiest-to-access form, but that's true not
only for you, but anyone else who rummages in your box.</p>
<p>I would suggest at least 2 locations. They should be far enough apart that
they're unlikely to suffer the same fate in the event of a disaster.</p>
<h2>Software</h2>
<p>I'm a big fan of paid-for backup software. Here are some specific packages:</p>
<p><strong>CrashPlan</strong> CrashPlan is best known as a service for backup. They also
provide hosted enterprise solutions, which allows you to have servers at their
own location. The software also can be used to backup to any location that
there is another crash plan user. This means that you and your body can provide
storage space for each other's backups, guaranteeing that if one of your houses
goes up in flames there is a copy of the data at the other house.
That's a neat feature, although I've never used it.
Generally speaking I have found the crash plan software to be reliable and those
support services to be adequate. I can comfortably suggest their service to
most users for off-site backup as it provides significant encryption and their
systems do constant reliability checks on the stored data.</p>
<p><strong>Time Machine</strong> Time Machine is Apple's built in backup software for many
versions of OS X. It provides version storage as well is very simple
administration, and can be used easily with an externally connected hard drive.
Of course, it's not very useful for off-site backup. However, for local backup
it is easy to set up and easy to restore data from.</p>
<p><strong>BRU Server</strong> I use BRU Server in daily use at ClueTrust for the machines in
the hosting center. It's not the prettiest software package (by a mile),
but it works and works reliably. We had a serious hardware event over the first
of the year, and it came through with flying colors. The software itself
installs on a server and then individual agents are installed on each client
machine. Agents are available for just about any operating system that you can
imagine including more varieties of UNIX and I thought even existed anymore.
I don't have much experience with the Windows agent, but the Mac agents work
well, and the UNIX agents also function just fine.</p>
<p><strong>SuperDuper!</strong> SuperDuper is a package that clones hard drives on the
Mac from one device to another. This provides you with a completely bootable
version of the device as of the time that you created the clone.
These backups are exact duplicates, and that means that you don't
get to go back and look at previous versions of the files.</p>
<p><strong>Retrospect</strong> Historically (in the old days), I used Retrospect,
which went down hill significantly when the Dantz was acquired by EMC.
The software product was spun back out into Retrospect, Inc. in November of
2011, and the word is that it has improved markedly since then.
I gave it a try again once, but have not used it in production,
nor have I tried recent versions, but reliable sources say that it is
getting better.</p>
<h2>Encryption</h2>
<p>When possible, do this. It's especially important when keeping data off-site
to make sure that data is encrypted using strong encryption and with keys that
are only available to you. This is possible with some services like CrashPlan,
which allow you to designate your own keys and is possible when you store data
encrypted on your own hard drives. Any data which is intentionally taken off-
site should be stored in some encrypted form. Keep in mind that if you
designate your own keys, you are going to have to safely store these keys in a
manner that they will not be lost by whatever event causes your data to be
lost.</p>
<h2>In Conclusion</h2>
<p>It doesn't really matter as much how you decide to back up your data, it just
matters that you do back up your data. If there's something that you care
about, back it up. If you care about the data being secure, encrypted it. If
for some reason you believe you care about the data and you don't care about
being secure, think again.</p>
FirstToDisclose.org repository for invention disclosure2013-03-29T04:22:00-04:002013-03-29T04:22:00-04:00Gaige B. Paulsentag:www.gaige.net,2013-03-29:/firsttodiscloseorg-repository-for-invention-disclosure.html<p>Launched just ahead of the first move by the US to switch from first-to-invent
to first-to-file, a new site,
FirsttoDisclose.org has launched, with the idea
of putting your materials in the public view in order to limit other people
from attempting to claim patent protection for ideas you are …</p><p>Launched just ahead of the first move by the US to switch from first-to-invent
to first-to-file, a new site,
FirsttoDisclose.org has launched, with the idea
of putting your materials in the public view in order to limit other people
from attempting to claim patent protection for ideas you are using. I'm not
a lawyer, but from what I can see, this is mostly useful for forcing ideas to
be disclosed for public use (i.e. intent to eventually place in the public
domain).</p>
<p>It's an interesting idea, and has gotten some press coverage, but I'm not sure
how much traction it will get, especially in this format.</p>
<p>Some further information about the move toward First-to-File is presented in
an article by two IP lawyers from Steptoe & Johnson LLP:
<a href="https://web.archive.org/web/20130320011803/http://www.steptoe.com/assets/htmldocuments/DJ%20-%20PATENT%20LAW%20-%20KOVELMAN%20PETERSON%20-%202011.pdf">Patent law creates a first-to-disclose system</a>
(PDF), which discusses exceptions in the prior art handling mechanism in the
new law, something that leaves some differentiation between the US system and
those of other first-to-file systems. It also describes some interesting
scenarios where the US system might lead people to intentionally disclose
earlier in some fields, which would preserve their rights in the US, and
possibly create problems outside of the US. There have been similar
problems in the past with PPAs here in the US and foreign rights.</p>
<p>According to the FAQ (confirmed by a
<a href="https://web.archive.org/web/20151017113341/http://www.blipclinic.org/2013/03/brooklyn-law-students-launch-firsttodisclose-org-in-anticipation-of-new-america-invents-act-patent-priority-rules/">press release</a> from the clinic), the site was set up by members of the
Brooklyn Law Incubator and Policy Clinic (BLIP), and is registered to an
individual with an address in Brooklyn.</p>
<p>As of today, there is one test disclosure up on the site, and no information
on the number of registrants.</p>
<p><em>Ed Note</em>: Sadly, this site went away and is now being used by a law firm for
other purposes. So much for a repository of disclosures.</p>
New server, new design2013-03-28T08:08:00-04:002013-03-28T08:08:00-04:00Gaige B. Paulsentag:www.gaige.net,2013-03-28:/new-server-new-design.html<p>Hi folks. We're back on the air with a new server and a new design.
Hopefully this less noisy design is a bit more palatable. Any links to the
old site will cease to function today, but there weren't many anyway
(according to Google Webmaster Tools), but we have preserved …</p><p>Hi folks. We're back on the air with a new server and a new design.
Hopefully this less noisy design is a bit more palatable. Any links to the
old site will cease to function today, but there weren't many anyway
(according to Google Webmaster Tools), but we have preserved all of the
content.</p>
<p>My plan is to post more in 2013 than I did in 2012, which I believe I have
already achieved by posting a single article.</p>
<p>Cheers for now, and set the new RSS feed to keep up to date.</p>
Where'd my darned flash go!2011-12-27T14:37:00-05:002011-12-27T14:37:00-05:00Gaige B. Paulsentag:www.gaige.net,2011-12-27:/whered-my-darned-flash-go.html<p>I received a question this afternoon from my cousin about the amount of free
flash in her MacBook Air and figured that the answer would probably be useful
to others as well. Note that none of this is officially from Apple, so it
might be wrong, but I have had …</p><p>I received a question this afternoon from my cousin about the amount of free
flash in her MacBook Air and figured that the answer would probably be useful
to others as well. Note that none of this is officially from Apple, so it
might be wrong, but I have had quite a few SSDs and the vast majority of it is
correct, or at least a jumping off point.</p>
<p>So, my tech-savvy cousin sent me the output of "df -k" in terminal and
wondered: "Where's the rest of my hard disk space?" As background, she has a
MacBook Air with a 128GB Hard disk and the output of df is:</p>
<div class="codehilite"><pre><span></span><code><span class="go">Filesystem 1024-blocks Used Available Capacity Mounted on</span>
<span class="go">/dev/disk0s2 117649480 114646588 2746892 98%</span>
<span class="go">/devfs 179 179 0 100%</span>
<span class="go">/devmap -hosts 0 0 0 100%</span>
<span class="go">/netmap auto_home 0 0 0 100% /home</span>
</code></pre></div>
<p>There are a number of things that are at play in this.</p>
<p>First, 128GB is 128,000,000,000 bytes and the 1024 blocks are in 1024 byte
chunks. So, your 117,649,480 is 120.4730 billion bytes. Then, if you're
running Lion (which you should be), there will be the recovery partition,
which is hidden but absorbs about 1.3GB on my MacBook Pro.</p>
<p>The remainder is likely to be formatting slop and wear leveling space. The
latter is used to make sure that things don't go haywire as SSDs have some
peculiar requirements as to how data is written, etc.... this is one of the
reasons that Apple bought a flash memory controller manufacturer in Israel
late last week.</p>
<p>Any rate, as for the "private", that's the disk space that is required to make
your computer work well. The VM is what is called "virtual memory backing
store" and provides space for the computer to store stuff that should have
been in memory if your computer had 8 or 16 or 32GB of RAM. Without it, you
would have to quit more frequently out of programs in order to get other
programs to run. (Incidentally, there is no VM on the iPhone and iPad, because
they just kill off programs that are taking up memory in the background, but
that can't be done with Apps on the Mac very easily, since they were mostly
written a long time ago... the good news is that with Lion there is
infrastructure for apps to support this kind of behavior in the future). The
Sleep Image is what allows your computer to go into deep sleep and not use
any/much battery when you have the lid closed. Here you can be thankful you
have only 4GB of RAM, because my laptop eats 8GB for its sleep image....</p>
<p>So, in summary, this all looks pretty normal under the circumstances. You can
turn off the sleep image, but I would strongly suggest against it. As a
general rule, you want to keep between 8-16GB Free as a minimum on SSDs, and
30-50% free on rotating (old style) disks in order to maximize performance.
The more full your disk, the more rewrites you flash will take and the sooner
it will wear out.</p>
<p>In the end, as with all disks, it's a balance between performance and
longevity vs what you want to have with you. The trade-offs are different
between rotating media and solid-state, in that disks can wear out without
much writing, whereas an SSD that is read most of the time will have very
little "wear".</p>
Verb-first AppleScript commands2010-11-03T13:35:00-04:002010-11-03T13:35:00-04:00Gaige B. Paulsentag:www.gaige.net,2010-11-03:/verb-first-applescript-commands.html<p>This is a note for those of us who might run into this problem. When working
on some changes for Cartographica, I ran into some difficulty when using a
verb-first command that could take either a list of one type of user-defined
object or another type of user-defined object.</p>
<p>The …</p><p>This is a note for those of us who might run into this problem. When working
on some changes for Cartographica, I ran into some difficulty when using a
verb-first command that could take either a list of one type of user-defined
object or another type of user-defined object.</p>
<p>The list worked fine (once I realized that it was going to come in as a set of
<code>NSScriptObjectSpecifiers</code> that needed to be evaluated). However, I couldn't get
the <code>NSScriptCommand</code> to receive the call for the stand-alone object of a
different type. AppleScript kept complaining that the class in question didn't
respond to that message. Well, mostly right, except that the command, which
was specified elsewhere should have been taking care of that, or so I thought.</p>
<p>If you have an <code>NSScriptCommand</code>-based command that takes direct parameters, you
need the class to specify that it responds to that message, even if you do so
by adding an empty method descriptor in the <code>responds-to</code>. If <code>responds-to</code> isn't
there, it gets blocked up at a higher level.</p>
<p>So, the solution was simply adding an empty <code>responds-to</code> for the command that
I'd added and then leaving the work to the <code>NSScriptCommand</code>-derived class.</p>
<p>I hope this saves somebody some time.</p>
Flash, ubiquitous mediocrity2010-04-30T03:44:00-04:002010-04-30T03:44:00-04:00Gaige B. Paulsentag:www.gaige.net,2010-04-30:/flash-ubiquitous-mediocrity.html<p>So with all the wringing of hands and gnashing of teeth over Adobe Flash in
the last few days, I just wanted to make sure I got this little tidbit in.
When it comes to user experience, Adobe just doesn't understand. Their
"platform" of flash brings a mediocrity born of …</p><p>So with all the wringing of hands and gnashing of teeth over Adobe Flash in
the last few days, I just wanted to make sure I got this little tidbit in.
When it comes to user experience, Adobe just doesn't understand. Their
"platform" of flash brings a mediocrity born of the Internet of 5-10 years ago
into your browser window. Why? Because, with Flash you can build once and
deploy everywhere.... Wait a second, wasn't that Java? Have you ever used a
Java application? They just don't feel right on any platform and neither does
Flash. My troubles today began when I headed over to the Adobe site to give
them my bi-annual upgrade fee...</p>
<p>As painful as it can be to spend $600 every two years or so to keep your Adobe
products up to date, they still have some of the best professional products
out there, especially Photoshop, Illustrator, and InDesign. I tolerate
DreamWeaver, but less and less every year, especially since I have some very
talented people working the web work for me these days.</p>
<p>So, I headed on over to Adobe and got a big Flash window to start with.
Unnecessary animation, and with no video going 100% of one of my 16 cores
operating. Now, to be absolutely clear, there is <strong>no reason for this CPU
use</strong>. They have some vector animation going on and are sitting in a tight
loop while waiting for me to do something no the screen.</p>
<p>I continued on the purchase track, navigating the information about the
various types of upgrades and noting that you just can't get everything you
want without buying Master Suite, but as with last time, I skipped it as I
will likely not need the other features anyway.</p>
<p>Clicking on "Check Out," I went to another, slow-loading flash page. This
finally finished, but rather than do a narrative, I'm just going to list out
the things that just plain don't work well for me on the Adobe site, all due
to Flash:</p>
<ul>
<li>1Password secure login credential manager can't be used to log in- I had to go dig out my password. Thankfully, at least they support paste, but that was as far as I was able to automate my login process.</li>
<li>1Password secure credit card entry doesn't work either. Probably redundant to the first one, but it was highly annoying.</li>
<li>Scrolling with my magic mouse doesn't work. Yes, there's a very standard event handling API on the Mac for this, but apparently since it isn't part of the Adobe platform, my user experience as a Mac user is going to suffer. If this were an HTML5 Application, the scrolling and user experience would be appropriate for whatever platform I was running on, not some middle-of-the-road platform developed by Adobe that lowers every platform to the lowest common denominator.</li>
<li>Double-click selection behavior doesn't work. Double-click and drag doesn't do extended word selection like it does in every Macintosh Application.</li>
<li>And this is all without commenting on the extremely lame use of the Akamai (Java-based) download manager.... 3 different windows opening every time I add another file, and there were 7 of them..... lame.</li>
</ul>
<p>This is just a small list. When you are working in a Flash environment, things
just feel "wrong". It's not always easy to put your finger on it, although the
additional background CPU usage certainly contributes to some sluggishness on
the entire machine, not to mention in the Flash "application".</p>
<p>I understand the purpose of using Flash. For Adobe, it's a way to try and out-
do Java (develop once and deploy everywhere), which never worked that well for
the users (or for Sun for that matter). For designers, it's a way to develop
and application without having to hire a programmer. Adobe even describes it
as drag-and-drop programming. That's all well and good, but as a long-time
developer, I understand that the quality of a product is often based on the
amount of work that you put into it. I don't doubt that you can create
reasonable user experiences with Flash, but I don't think you can create them
easily. What Flash enables is piss-poor quality content creation for
everyone.... ubiquitous mediocrity.</p>
Apple replaces ADC with Mac Developer Program2010-03-05T04:15:00-05:002010-03-05T04:15:00-05:00Gaige B. Paulsentag:www.gaige.net,2010-03-05:/apple-replaces-adc-with-mac-developer-program.html<p>Wow! Apple certainly wants more people to sign up as Mac developers, that's
for sure. In changes made today, the ADC Premier and Select tiers have
disappeared and they have been replaced with Mac Developer through the current
developer portal (which has the iPhone developer sign-up as well). Mac and …</p><p>Wow! Apple certainly wants more people to sign up as Mac developers, that's
for sure. In changes made today, the ADC Premier and Select tiers have
disappeared and they have been replaced with Mac Developer through the current
developer portal (which has the iPhone developer sign-up as well). Mac and
iPhone are now $99/year each and doesn't include the hardware discount, which
has been an historical benefit of the more expensive developer program. More
importantly, you can't buy the Premier version, which includes your ticket to
the next WWDC, so you'll be fighting the crowd with everyone else.</p>
First media rant of the year2010-01-02T12:28:00-05:002010-01-02T12:28:00-05:00Gaige B. Paulsentag:www.gaige.net,2010-01-02:/first-media-rant-of-the-year.html<p>That didn't take too long. Dana Milbank of the Washington Post has a column
coming out tomorrow (which I won't link to because of my theory that this is
all about the flame bait and web hits). The article basically states that
Glenn Beck is more admired than The Pope …</p><p>That didn't take too long. Dana Milbank of the Washington Post has a column
coming out tomorrow (which I won't link to because of my theory that this is
all about the flame bait and web hits). The article basically states that
Glenn Beck is more admired than The Pope. Ah, the cherry-picking! So, the
original <a href="http://www.gallup.com/poll/124895/Clinton-Edges-Palin-Admired-Woman.aspx">poll</a>
(<a href="http://www.Gallup.com">Gallup</a>, based on 1025 surveyed adults,
estimated +/- 4% error rate) does seem to show that if you look at one
particular piece of it whilst ignoring everything else. However....</p>
<p>If you actually read the numbers, you see that in the Men category, everybody
below Barack Obama (for the aggregate general public of all political
persuasions) are literally within the margin of error of ZERO. That's right,
nobody but Barack Obama had more than 4% of the combined total, and only
George W Bush (the Republicans' #1 at 11% of Republicans) and Nelson Mandella
had more than 2% of the combined total.</p>
<p>Even when you go to the Republicans only, Glen Beck still had 3%. To me, this
looks like the Washington Post is paying their columnists for either the # of
comments or the # of hits on their web site, because that'd be about the only
reason to write such a reactionary piece of drivel.</p>
<p>And, I'm not even sure that you could argue that this is "good for the liberal
cause" as he likes to be. Most people will read maybe the headline and the
first paragraph, and they'll believe that this is true. Republicans will think
that Beck is well thought of by everybody but them, and Democrats will think
that all Republicans are idiots because they admire this guy. In the end,
neither is the case.</p>
<p>Today, we heard of the passing of Deborah Howell, the former Ombudsman of the
Washington Post. I can't say that she was a personal friend, but I did have
correspondence with her on a few occasions and she was one of the reasons I
thought the Post might recover...I can't help but think that I'm happy she
never had to read this drivel.</p>
Pogue on an author's view of DRM for books2009-12-19T04:58:00-05:002009-12-19T04:58:00-05:00Gaige B. Paulsentag:www.gaige.net,2009-12-19:/pogue-on-an-authors-view-of-drm-for-books.html<p>David Pogue (<a href="http://www.NYTimes.com">NY Times</a>) has written a blog entry
about his experience as an author <a href="http://pogue.blogs.nytimes.com/2009/12/17/should-e-books-be-copy-protected/">selling an ebook with no
DRM</a>...
and it <em>wasn't</em> the end of the world, or even his career as an
author. With the nook and a possible Apple tablet coming to contend with the
<a href="http://www.Kindle.com">Kindle …</a></p><p>David Pogue (<a href="http://www.NYTimes.com">NY Times</a>) has written a blog entry
about his experience as an author <a href="http://pogue.blogs.nytimes.com/2009/12/17/should-e-books-be-copy-protected/">selling an ebook with no
DRM</a>...
and it <em>wasn't</em> the end of the world, or even his career as an
author. With the nook and a possible Apple tablet coming to contend with the
<a href="http://www.Kindle.com">Kindle</a>, will the publishing industry realize that
this may just be another lock-in like Apple and the iPod (which basically
killed music DRM).</p>
<p>He goes on in the article to describe how scared the publishing industry is
about ebooks and the classic problems with DRM (hurts the people who are
honest, never really gets in the way of the people who aren't).</p>
<p>Then, he gets to the interesting part. It turns out that he and
<a href="http://www.OReilly.com">O'Reilly</a> got together to do an experiment and
offered one of his windows books online without DRM (but with pricing). The
book showed up quickly on file sharing sites, as was the case when he released
it in DRM'd form.</p>
<p>Speaking as somebody in the software industry, I feel for his concerns about
completely removing DRM. Without some form of reminder to people that the
books that they are reading are theirs and not free for the taking, it's
likely that even naturally honest people will find themselves "loaning" their
books out, only to have the recipient keep the books for a long time, which is
something that just doesn't happen with a physical book (at least not without
the original owner losing the ability to use the book while it's loaned out).</p>
<p>However, those of us in the software industry have long realized (except for
the saturation sellers, like Microsoft and Adobe) that the "copy protection"
portion of the software is there to keep the honest people honest and the
folks who are on the fence feeling bad, and it just doesn't have any effect on
those who are prone to copying... there's just no way to help those people
from their kleptomaniacal ways.</p>
<p>As a Kindle owner, I hope that the DRM for books is resolved soon, since I'm
very hesitant to buy books on the Kindle in its current form. However, I am
enjoying reading my PDF books that I purchased from places like
<a href="http://pragprog.com/">Pragmatic</a>, which offers DRM-free books in PDF (and now
I find out mobi) format that can be used on just about anything.</p>
Regret The Error: The Year in Media Errors and Corrections2009-12-17T04:03:00-05:002009-12-17T04:03:00-05:00Gaige B. Paulsentag:www.gaige.net,2009-12-17:/regret-the-error-the-year-in-media-errors-and-corrections.html<p><a href="https://web.archive.org/web/20100528163814/http://www.regrettheerror.com/2009/12/16/crunks-2009-the-year-in-media-errors-and-corrections/">Crunks 2009: The Year in Media Errors and
Corrections</a> is well worth a read to anyone who watches the
media at all. In particular, the "Correction of the Year" (a 9/11 vs 911
confusion) and the pointer to <a href="https://web.archive.org/web/20120705073113/http://j-source.ca/article/when-should-editors-unpublish-online-news-reports">When should editors "unpublish" online news
reports?</a> from the
<a href="http://j-source.ca/">Canadian Journalism …</a></p><p><a href="https://web.archive.org/web/20100528163814/http://www.regrettheerror.com/2009/12/16/crunks-2009-the-year-in-media-errors-and-corrections/">Crunks 2009: The Year in Media Errors and
Corrections</a> is well worth a read to anyone who watches the
media at all. In particular, the "Correction of the Year" (a 9/11 vs 911
confusion) and the pointer to <a href="https://web.archive.org/web/20120705073113/http://j-source.ca/article/when-should-editors-unpublish-online-news-reports">When should editors "unpublish" online news
reports?</a> from the
<a href="http://j-source.ca/">Canadian Journalism Project</a> were very
interesting.</p>
AT&T's complaining about iPhone users2009-12-13T06:51:00-05:002009-12-13T06:51:00-05:00Gaige B. Paulsentag:www.gaige.net,2009-12-13:/atts-complaining-about-iphone-users.html<p>I'm an iPhone user... my wife (Hi, Carol!) is an iPhone user, and I even have
an iPhone set up for development purposes that doesn't get used for anything
else (despite the fact that we pay for it monthly). I also have an AT&T
Data card, for use when …</p><p>I'm an iPhone user... my wife (Hi, Carol!) is an iPhone user, and I even have
an iPhone set up for development purposes that doesn't get used for anything
else (despite the fact that we pay for it monthly). I also have an AT&T
Data card, for use when hot spots are either unavailable or too annoying.
Generally speaking, I've had the same experience as most AT&T users in the
DC area, "meh". But, this latest complaining from AT&T about iPhone data
usage has gotten me a little hot under the collar.</p>
<p>According to widespread reports, AT&T's CEO (Ralph de la Vega) has been
complaining that those damnable iPhone users have created a situation where 3%
of the users are using 40% of the data on his network, and that's the reason
why his suddenly-very-popular-not-because-of-its-stellar-3g-coverage network
is bogged down.</p>
<p>Well, Ralph, I'm here to tell you that you're extracting a heck of a lot of
money from me for very little data usage. Carol & I use our iPhones pretty
heavily, as far as I'm concerned, both in the house and out. But, I've got
gone and looked at it, and we're averaging about 150MB/month in data usage
each. For this privilege, we pay the iPhone data surcharge (that doesn't cover
SMS any more) of $39.95 per month (that's in excess of our charges for voice
usage).</p>
<p>Comparison time here. I mentioned before that I have a data card, which has a
5GB limit and costs me $60/month to use. Based on the relative price, I would
expect that the iPhone user is paying for 2/3 of that 5GB, or about
3.33GB/month. Based on our usage (total of 300MB/month for 3 phones), we are
paying 39.95*3/300MB or $0.40/MB for our data usage. Yes, if the use
increased, we wouldn't pay more, but to put this into perspective, just on my
not-very-oft-used data card, I use an average of about 300MB/month. At
$60/month, that means I'm paying $60/300MB or $0.20/MB for data usage, or
literally half of what we're paying for our iPhone usage.</p>
<p>So, I don't see where AT&T is getting screwed here.</p>
<p>However, in reality, they haven't claimed anything other than 3% of the users
are using 40% of the data. This is pretty normal for ISPs. Since time
immemorial, ISPs have griped about the disproportionate use of the high-end
users over the low end. What they don't complain about is the 97% of users who
get crammed into the 60% usage. As a matter of fact, I don't ever remember
having an ISP complain that they were charging the low-end users too much...
nor that they didn't want the high end users touting their products, just that
the high end users were "unfairly using the resources", which is pure,
unmitigated greed.</p>
<p>Now, some of you may be saying that I should be more sympathetic to ISPs....
the heck with that, I've been an ISP, I've made money being an ISP, and I've
been the part owner of ISPs in 16 countries around the world, including Japan,
the Philippines, Canada, the US, Brasil, Chile, Germany, France, Switzerland,
Uruguay, Argentina, Panama, just to name some. In the end of the day, every
ISP gripes about data usage, but none of them complain about the word-of-mouth
from the power users and nobody sees usage patterns that are any different.
Data use in the ISP industry is progressively up and to the right. People
consume more data every day than the day before. ISPs don't have the luxury of
sitting still like phone companies of old and letting their ancient equipment
languish, they have to innovate and upgrade or they will be surpassed by their
competition.</p>
<p>In the old days, in America, we used to call this competition and call it a
good thing. We used to discuss how capitalism and competition was good for
customers and good for international stature, as it kept us competitive in the
light of everything going on in the world. However, I guess that's not good
for America any more. What seems to be good for America (at least according to
AT&T) is to complain about their customers until they stop using what
they're paying for and sit down and take it like a child being told what to
do. Sounds a bit like communism to me: everybody should use the same data so
that the centrally-planned AT&T can move slowly and with plenty of
bureaucracy.</p>
Aperture and laptops2009-10-06T14:52:00-04:002009-10-06T14:52:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-10-06:/aperture-and-laptops.html<p>For those of you digital photography enthusiasts who use Aperture, here's a
little bit of info on how I deal with going out of town with my laptop, but
wanting to do the "real work" on my desktop when I get home.</p>
<p>It turns out that if you are careful …</p><p>For those of you digital photography enthusiasts who use Aperture, here's a
little bit of info on how I deal with going out of town with my laptop, but
wanting to do the "real work" on my desktop when I get home.</p>
<p>It turns out that if you are careful, you can save yourself a lot of time and
effort. I have a new idea that I haven't tried yet, that might have some cute
results for working on the home network, but that is a more involved test that
I'll do later.</p>
<p>Right now, what I do is import the photo files into my Photos directory
manually (usually a subdirectory), and then import those files by reference
(don't copy the data) into a new Aperture Project on the laptop. This way, I
can play around with them and sort them prior to getting back to my main
computer.</p>
<p>When I return home, I then use the Export... command to export the Project
created above, taking care not to let it consolidate the data into the project
file. I then import this file over AppleShare by mounting the laptop disk on
the desktop machine. This imports the important data and leaves the files on
the laptop, where they are still available.</p>
<p>Once I'm happy with that, I use the Consolidate command to consolidate the
photos into the vault on my desktop machine, taking care to copy, not move,
the data. This way, the project on my laptop is still completely functional
until I decide to delete the data.</p>
<p>Once I'm satisfied that everything is moved over, I then delete the sub-folder
in the Photos directory and get my disk space back on the laptop.</p>
My take on Macintosh security2009-09-03T05:19:00-04:002009-09-03T05:19:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-09-03:/my-take-on-macintosh-security.html<p>Ah, a new release.... must be time for another slew of articles aimed at
getting press and money for the "security" folks out there. For those of us
with Macintoshes, here is my take on the whole Macintosh virus situation.</p>
<p>Every time a new OS release comes out, a whole …</p><p>Ah, a new release.... must be time for another slew of articles aimed at
getting press and money for the "security" folks out there. For those of us
with Macintoshes, here is my take on the whole Macintosh virus situation.</p>
<p>Every time a new OS release comes out, a whole mess of security
"professionals", especially those with recent books (such as Miller's <a href="http://www.amazon.com/gp/product/0470395362?ie=UTF8&tag=cartographica-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0470395362">The Mac
Hacker's Handbook</a>),
are being interviewed by every Tom, Dick, and Harry, and repeat the same
drivel that we've been hearing about Macintosh security for years, which
basically amounts to:</p>
<p><em>Oh yeah? Well, if more Macintoshes were sold, then there'd be a lot more
viruses for the Mac, I tell you.... just you wait!</em></p>
<p>Now, it may well be true that if there were more Macs out there, there would
be more reason to go after the Macintosh and it would tend to lead people to
write more viruses for them. It may also not be true, and I've <em>never</em> seen
any indications that there is a statistical basis for this complaint.</p>
<p>However, let's take for the moment that it's a possibility and start looking
at the kinds of exploits that tend to show up for the Macintosh in these
articles. Generally speaking, and I'm not going to cite individual articles
here because I haven't done a complete statistical analysis of them, the kinds
of exploits that show up for the Macintosh are trojan horses, a class of
malicious software that the user downloads and runs or installs. Once you've
done that, you're open to a number of potential problems, including the
stealing of data and the deletion of files that are not protected.</p>
<p>There are 2 key take-aways about trojan horses on the Mac: first, they are not
the same as viruses; and second, they are limited in what they can do to your
system <strong>unless you give them power</strong>. Now, this part in bold is important. If
you download a questionable piece of software from the Internet (or any
software for that matter, since most really don't need this facility) and the
software prompts you for a password to your system during the installation
process, you should be seriously considering saying "no". If you say yes, you
do not have any granular control of what it might do to your system, as you
have provided it with escalated privileges to access all data and services on
your Macintosh.</p>
<p>Here are a few other things that make a big difference to Macintosh users: no
in-the-wild viruses. There are basically no programs that exist today that can
infect Macintoshes without the user taking specific action (opening a program
in particular). Through the use of Quarantine, which has been around since
Leopard, Apple tries to warn you the first time you open a piece of software,
telling you where it was downloaded from asking you if you're sure you want to
run it. It only happens the first time you run each program, so it doesn't
provide an overwhelming number of "are you sure" dialogs.</p>
<p>Once you install a program on one Macintosh, the likelihood of it spreading
virally (without you or the user of the computer specifically starting the
program in question) is really, really low. I say really, really low, because
there were some programs that managed this feat before Leopard due to hiding
executables in what looked like data files. However, quarantine makes that
virtually impossible these days.</p>
<p>Most importantly, the kinds of worms that have infected Windows and other
systems over the years (a worm being a particularly vicious type of malware
that makes its entrance behind the scenes, infects the computer and uses it as
a jumping off place to infect more), have been almost absent from the Mac
(there was a report of one in 2006/2007 using Bonjour as a vector, but that
was patched by Apple on all affected systems and the worm appeared to only
show up after that problem was disclosed).</p>
<p>People can argue until their blue in the face about why Macs tend to have a
lot less trouble than PCs. Frankly, the amount of open administrative software
that lies on (especially older) Windows machines is a good portion of the
problem here. For years, Windows 2000 and other versions had the ability for
network administrators to broadcast a message to every user on a network that
was then displayed on their screens. This was a horrible idea, since it had
absolutely no security whatsoever involved in it and basically allowed anyone
with knowledge of your network address to send a message to your screen that
popped up as if it were from the OS. To make matters worse, there were
security problems with the program that put up the window and they were
exploited to deliver worms and other viruses on the Windows platform. This is
not an isolated case, either.</p>
<p>Architecturally, there's definitely more that Apple can do about security on
the Macintosh and I hope that we continue to see the kind of sandboxing that
is being used by Apple on the iPhone slowly creep its way into the Mac. By
using this judiciously, they could keep only authorized programs from doing
things on the system and they could make a much better permissions model for
the otherwise-dangerously all or nothing approach that the installers tend to
take these days. I'd love to see something along the lines of an installation
dialog for VMWare (as an example) that requests permission to "add kernel
extensions and startup items" and then have the OS grant just permissions to
install items in those places. More importantly, for programs that use the
installer just to put things into special locations, such a scheme could
prevent them from doing other things behind the scenes (like installing kernel
extensions) without your knowledge. I know I'd think twice if a graphics
program requested permission to install a kernel extension.</p>
<p>But, for the time being, the Macintosh is a pretty safe platform, as long as
users are vigilant. Keep up to date on your software updates and don't run
programs with questionable pedigrees.</p>
<p>NOTE: Today's <a href="http://www.Wired.com">Wired</a>
<a href="http://www.wired.com/gadgetlab/2009/09/security-snow-leopard">article</a> pretty
much caused this article to be written. I have to say that you must admire a
magazine that continues such superlative reporting as telling us that "In Snow
Leopard, Apple has added security enhancements including Executive Disable"...
executive disable? Sounds like something you'd use in a bad movie to remove
your competition, did you mean Execute Disable (XD), a technology that's been
around for years and was one of the most touted security features of the last
3 generations of processors? Oh, you know, that whole accuracy thing isn't
important. Wonder how well you did on the other facts? Probably about the
same, interview a couple of guys who are shilling a book and reprint their
stuff as well as whatever you can find in a quick Google search. No offense to
Google. For more humor, the next line: "Apple also added hardware-enforced
Data Execution Prevention" is basically a reference to the <em>Exact Same
Technology</em>. Curiously, Apple's only technology mention is of "hardware-based
execute disable for heap memory", which I'll note doesn't mention disabling
executives at all!</p>
Building IBPlugins under Leopard and Snow Leopard2009-08-31T10:40:00-04:002009-08-31T10:40:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-08-31:/building-ibplugins-under-leopard-and-snow-leopard.html<p>This is a pretty esoteric topic, but I ran into it and maybe the google will
help somebody find my solution before they waste too much time. I have a
custom control with a custom IBPlugin which we use in Cartographica. The
IBPlugin compiles in all 4 binary modes, but …</p><p>This is a pretty esoteric topic, but I ran into it and maybe the google will
help somebody find my solution before they waste too much time. I have a
custom control with a custom IBPlugin which we use in Cartographica. The
IBPlugin compiles in all 4 binary modes, but I restricted it to ppc and i386,
because the IBFramework which you have to link to doesn't support 64-bit under
Leopard. Enter Snow Leopard...</p>
<p>So, under Snow Leopard, IB (Interface Builder) is a 64-bit application and
both IB and <code>CompileXIB</code> complain that the plugin doesn't have the right number
of bits in it. So, I need to build 64-bit, which wasn't difficult.
Unfortunately, the hard part is that if you try to link the 64-bit version
under Leopard, it chokes.</p>
<p>This wouldn't be a huge problem, since I'm building under Snow Leopard (even
the 10.5 compatible version) right now, but for safety's sake and because I
haven't upgraded our buildbots yet, I run the automated tests on Leopard.</p>
<p>My solution was rather simple, I pointed the target at a user-defined variable
and then built for all architectures on Snow Leopard. For the leopard compiles
(which are all done by script anyway), I just put in a define that overrides
that variable at compile time and it worked fine.</p>
<p>Since this is built as a part of the multi-stage automated build process using
dependencies, I couldn't just set up something simple, so in my case I had to
make sure that the variable in question wasn't used by anything else.</p>
Alton Brown hacks the Kitchen2009-08-26T05:40:00-04:002009-08-26T05:40:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-08-26:/alton-brown-hacks-the-kitchen.html<p>Cool <a href="http://gizmodo.com/5344726/alton-brown-safe-and-scary-kitchen-hacks">piece</a>
on <a href="http://www.Gizmodo.com">Gizmodo</a> about Alton Brown's favorite Kitchen hacks.</p>
Snow Leopard Releases Friday2009-08-25T14:29:00-04:002009-08-25T14:29:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-08-25:/snow-leopard-releases-friday.html<p>Many of you have already seen that Snow Leopard (OS X 10.6) will be releasing
on Friday. I've been running it as my primary OS on my laptop since the WWDC
in June and look forward to getting a real install on there. You're probably
also aware that PPC …</p><p>Many of you have already seen that Snow Leopard (OS X 10.6) will be releasing
on Friday. I've been running it as my primary OS on my laptop since the WWDC
in June and look forward to getting a real install on there. You're probably
also aware that PPC machines will not run 10.6 (and beyond), so this upgrade
is Intel Macintosh only. Here is roadmap through the upgrade products.</p>
<p>First of all, this cat is coming pretty cheap for those who already have
Leopard (either through purchasing a machine with Leopard on it, or upgrading
in the past). The MSRP for Snow Leopard is $29 ($49 for a family pack).</p>
<p>Apple's other option is a pretty good deal for pre-Leopard Intel users who are
also behind on iLife and are interested in iWork. You can get a bundle (called
the Mac Box Set) for $169 MSRP including Snow Leopard, iWork '09 and iLife '09
($229 for the family pack of all three).</p>
<p>Of course, if you order direct from Apple you can get it on Friday. However,
Amazon (and other retailers) are providing a financial incentive to ordering
from them.</p>
<ul>
<li>Snow Leopard single-user upgrade $24.99</li>
<li>Snow Leopard Family Pack upgrade $43.99</li>
<li>Mac Box Set (with Snow Leopard) $149.99</li>
<li>Mac Box Set Family Pack (with Snow Leopard) $199.99</li>
</ul>
<p>And for those of you on the server side, there's been a big drop in price from
the previous $999 and $499 prices for Unlimited and 10-user. Snow Leopard
Server will only be offered in Unlimited and the price has been moved to $499
(the previous 10-user price). Amazon has it for $444.99.</p>
Timing a UPS2009-08-21T09:39:00-04:002009-08-21T09:39:00-04:00Gaige B. Paulsentag:www.gaige.net,2009-08-21:/timing-a-ups.html<p>In line with my BackUPS/SmartUPS story earlier today, I wanted to say a little
something about how I "watched" it.</p>
<p>First, I am very pleased with my
<a href="http://www.amazon.com/gp/product/B00009MDBU?ie=UTF8&tag=cartographica-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=B00009MDBU">KILL-A-WATT</a>
for figuring out the real load (not needing to trust the meter on the UPS,
which was reasonably accurate), so I …</p><p>In line with my BackUPS/SmartUPS story earlier today, I wanted to say a little
something about how I "watched" it.</p>
<p>First, I am very pleased with my
<a href="http://www.amazon.com/gp/product/B00009MDBU?ie=UTF8&tag=cartographica-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=B00009MDBU">KILL-A-WATT</a>
for figuring out the real load (not needing to trust the meter on the UPS,
which was reasonably accurate), so I know that my equipment was using between
290 and 340 watts.</p>
<p>Next, I pulled the BackUPS out into a well-lit area and plugged it in (to top
it off). Then, I plugged 3 100-Watt lamps into it to generate about 300W of
power use.</p>
<p>Then came the realization that "there's a technology for that"... originally I
was going to sit and star at the meter just like I did timing out the SmartUPS
1500 yesterday afternoon (with a bit more reason, since there's no readout on
it). However, in this case, there was a screen and I figured I'd use my
camcorder to grab a movie of it running down so I didn't have to take readings
manually or watch the clock. It worked flawlessly.</p>
<p>In the end, I just waited for the lights to go out, wound back the "tape"
(it's a memory-based camcorder, so there really isn't any tape in it), and
paid attention to the time stamp at each significant event. Then I took a few
minutes to write it up and sent it off to APC.</p>
<p>Of course, they want me to "recalibrate" it and try it again... so I agreed,
since I now have a way to make that only expend wall-clock time.</p>