DevOps experience

Tailscale. A VPN you forget is even there.

2026-05-10T00:00:00Z

There’s a two-part series on this blog from 2017 about setting up Cisco ASA with FreeRADIUS and two-factor authentication. The respectful suggestion in 2026 is: don’t do any of that anymore.

Specifically, if you are a small team or an individual and you need a VPN, you almost certainly do not need to operate a VPN gateway. The whole category got reinvented around WireGuard, and the practical face of it for most people is Tailscale.

What Tailscale is

Tailscale is a control plane on top of WireGuard. WireGuard itself is a kernel module (on Linux) or a userspace library, and a tiny, beautiful protocol. The hard part of WireGuard, historically, has been key distribution, peer discovery, and NAT traversal. Tailscale handles those.

You install the client on every device you own — laptop, server, phone, NAS, whatever — log into a SSO provider, and they all show up as machines on a private network with stable IPs in the 100.64/10 range. Connections between any two of those machines go peer-to-peer over WireGuard, with the Tailscale coordination server only used for handshake.

The user experience is: ssh prod-bastion and you’re in. There’s no VPN client to start, no tunnel to bring up, no kill switch toggle. Tailscale is running in the background. That is the whole product.

Setting it up on a server

For comparison, the 2017 Cisco posts were about 3,000 words of config between them. Here is the 2026 equivalent for a Linux server:

$ curl -fsSL https://tailscale.com/install.sh | sh
$ sudo tailscale up --ssh

That is it. The --ssh flag opts the host into Tailscale SSH, which uses your Tailscale identity instead of static SSH keys. ACLs are configured centrally and applied across all your machines.

For an unattended server you want non-interactive setup, which uses an auth key:

$ sudo tailscale up \
    --authkey=tskey-auth-… \
    --hostname=prod-bastion \
    --ssh \
    --advertise-tags=tag:server

Auth keys can be ephemeral (the machine disappears from the tailnet when it’s offline for a while), reusable, or one-shot. For ECS / Kubernetes workloads I use ephemeral, tagged keys generated from Terraform.

ACLs are actually good

The ACL file is a single JSON / HuJSON document. It is short, readable, and lives in version control. A starter that covers most teams:

{
  "tagOwners": {
    "tag:server":  ["group:admins"],
    "tag:db":      ["group:admins"],
    "tag:dev":     ["autogroup:member"]
  },
  "groups": {
    "group:admins": ["artem@example.com"]
  },
  "acls": [
    // everyone can talk to dev boxes
    { "action": "accept", "src": ["autogroup:member"], "dst": ["tag:dev:*"] },
    // only admins can talk to db boxes
    { "action": "accept", "src": ["group:admins"],     "dst": ["tag:db:*"]  },
    // server-to-server within the tailnet
    { "action": "accept", "src": ["tag:server"],       "dst": ["tag:server:*"] }
  ],
  "ssh": [
    { "action": "check", "src": ["autogroup:member"], "dst": ["autogroup:self"], "users": ["autogroup:nonroot"] }
  ]
}

The "check" action on the SSH rule means the connection still requires you to re-authenticate via the IdP — basically a soft 2FA prompt — before SSH lets you in. The first time you see this in production it feels like cheating.

What about my private network?

If you have a VPC or an on-prem subnet that you want reachable from the tailnet, you put a Tailscale client on one box in that subnet and tell it to advertise the routes. The Cisco ASA equivalent of this used to be a series of routing tables and IPSec phase-2 selectors. Now it is one flag:

$ sudo tailscale up \
    --advertise-routes=10.0.0.0/16 \
    --ssh

Then in the admin console you approve the advertised route once. Done.

The honest trade-offs

The coordination plane is hosted by Tailscale Inc. (You can self-host it with Headscale, which is fine, but you give up the polish.)
For a small team Tailscale is free. Above 3 users you’re on a paid plan. For a personal project this never matters.
Throughput is limited by WireGuard itself, which is fast in absolute terms but obviously less fast than no tunnel at all. For everything I do — SSH, web admin, occasional file copy — it does not matter.

That is the recommendation. There is no good reason to run a corporate VPN gateway by hand in 2026 unless something specific forces your hand. The 2017 Cisco posts are preserved here for historical interest — and as a reminder that the right answer to a problem changes if you wait long enough.

GitHub Actions for AWS deployments. A small, sane setup.

2026-04-08T00:00:00Z

This is the GitHub Actions starter I want everyone on my team to use for AWS deploys. No long-lived access keys, no plaintext secrets, no magic. The whole thing is about 60 lines of YAML.

This is not a rerun of the older Lambda-and-GitHub post on this blog. That one was about reacting to GitHub webhooks from AWS. This is the inverse: deploying to AWS from GitHub. Different direction, different problem.

The old way

For years the standard pattern was:

Create an IAM user.
Generate an access key for it.
Paste the key into a GitHub repository secret.
Hope the key never leaks. Rotate when you remember.

It worked. It was also the source of basically every “our production access keys ended up in a public repo” incident.

The new way: OIDC

GitHub Actions can be an OIDC identity provider. Same idea as IRSA on EKS — your workflow gets a short-lived JWT signed by GitHub, AWS trusts that JWT under specific conditions, the workflow assumes a role with no long-lived secret ever existing.

The pieces:

Register token.actions.githubusercontent.com as an OIDC provider in IAM.
Create an IAM role whose trust policy allows assumption from that provider, scoped to your repo and (ideally) a specific branch or environment.
In the workflow, call aws-actions/configure-aws-credentials@v4 with role-to-assume. No secrets.

The trust policy

The least-permissive version pins the role to a single repo and a single ref:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::1111…:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:yourorg/my-service:ref:refs/heads/main"
      }
    }
  }]
}

For pull-request previews you would broaden the sub pattern, for example:

"token.actions.githubusercontent.com:sub": "repo:yourorg/my-service:pull_request"

And for a tag-driven release flow:

"token.actions.githubusercontent.com:sub": "repo:yourorg/my-service:ref:refs/tags/v*"

Whatever you do, don’t use repo:org/*. That trusts every workflow in every repo of the org. People do this. People shouldn’t.

The workflow

This is the whole thing. .github/workflows/deploy.yml:

name: deploy

on:
  push:
    branches: [main]

permissions:
  id-token: write       # required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Assume deploy role
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::1111…:role/github-deploy-my-service
          aws-region: eu-west-1

      - name: Build & push image
        run: |
          ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
          REPO=$ACCOUNT.dkr.ecr.eu-west-1.amazonaws.com/my-service
          aws ecr get-login-password --region eu-west-1 \
            | docker login --username AWS --password-stdin $REPO
          docker build -t $REPO:${{ github.sha }} .
          docker push $REPO:${{ github.sha }}

      - name: Deploy
        run: |
          aws ecs update-service \
            --cluster prod \
            --service my-service \
            --force-new-deployment

Two things worth highlighting:

The permissions: block at the top is mandatory. Without id-token: write the OIDC token isn’t minted, and you’ll get a confusing 403 from STS.
environment: production hooks this job into GitHub’s environments feature, which lets you require approvals, restrict deploys to specific branches, and have per-environment OIDC subjects. Use it.

The bit I keep forgetting

GitHub Actions runners can be slow to publish their token to STS during high-load periods. configure-aws-credentials handles the retry for you, but if you write the assume-role call by hand using the AWS CLI you’ll occasionally see InvalidIdentityToken on the first try. The action is doing more for you than it looks like.

That’s it. Sixty lines, zero long-lived secrets, scoped to one repo and one branch. If your AWS deploy workflow still uses access keys, this is the migration to do this quarter.

OpenTofu vs Terraform. Should I switch?

2026-03-15T00:00:00Z

Terraform isn’t MPL anymore. OpenTofu is the community fork. If you’re responsible for a Terraform codebase you probably want a considered answer to “should I switch.” Here is mine.

What actually happened

Quick recap, with as little drama as possible:

August 2023: HashiCorp re-licensed Terraform (and all their other products) from MPL 2.0 to the Business Source License. The TL;DR: still free for most uses, not free if you compete with HashiCorp’s commercial products.
September 2023: A coalition of vendors and individuals announced a fork. It was renamed to OpenTofu and adopted by the Linux Foundation in early 2024.
2024–2025: OpenTofu released 1.6, 1.7, 1.8, 1.9. It tracks Terraform’s features pretty closely but has shipped some of its own — early-evaluation variables, encrypted state files, the for_each in provider blocks.

That is the entire substance of the “drama.” The boring practical question is whether the syntax you already write still works.

The practical answer

For almost everyone the answer is: yes, both still work, OpenTofu is a drop-in replacement, you can switch the binary and your code is fine. I have done the switch on two reasonable-sized codebases. Both times it went like this:

$ brew install opentofu       # or whatever your package manager is
$ tofu init
$ tofu plan
$ tofu apply

That is the migration. The state file format is compatible. terraform_remote_state still works. Modules from the registry still install. The same providers from registry.terraform.io can be used, and OpenTofu also has its own registry at registry.opentofu.org mirroring most of them.

The minor wrinkle is the binary name. tofu instead of terraform. If you have CI scripts or developer workflows hardcoded to terraform plan, that’s a search-and-replace. Some teams alias terraform to tofu on shared runners during the transition. That works, but I would rather rip the band-aid off.

Should you switch?

The reasons to switch:

License clarity. If your company has a clear policy about not using BSL software in production, the decision is made for you.
You want the new features. Encrypted state out of the box is the one I have actually used.
You don’t want to depend on a single vendor’s commercial roadmap for what is, by now, very foundational software.

The reasons not to switch:

You use HCP Terraform / Terraform Cloud / Sentinel policies. That ecosystem is HashiCorp-only. OpenTofu does not currently aim to clone it.
You use a specific recent feature in stable Terraform that OpenTofu has not yet shipped. The gap is shrinking but they are not exactly the same software.
Your team has zero appetite for tool-name changes — which, fair enough, is sometimes the only reason that matters.

Running both

You don’t actually have to choose. The two binaries can operate on the same code, sometimes even on the same state, as long as you avoid features that exist in only one of them. I have a small CI matrix on one project that runs terraform fmt -check and tofu fmt -check, and terraform validate and tofu validate, as a smoke test. It catches accidental drift.

jobs:
  lint:
    strategy:
      matrix:
        tool: [terraform, tofu]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        if: matrix.tool == 'terraform'
      - uses: opentofu/setup-opentofu@v1
        if: matrix.tool == 'tofu'
      - run: ${{ matrix.tool }} fmt -check -recursive
      - run: ${{ matrix.tool }} validate

What I actually do

I run OpenTofu locally and in CI for everything new. I have not migrated a couple of older systems that are tied to a Terraform Cloud workspace. I don’t feel any urgency about it. The point of infrastructure-as-code was always that the code was the durable artifact, not the binary that interprets it.

The thing I would not do is wait for the dust to settle. The dust has settled. OpenTofu is here, it is fine, the licensing is clean. If you have a quiet afternoon, switch and stop thinking about it.

IAM roles for service accounts on EKS. A small primer.

2026-02-20T00:00:00Z

One of the small things in EKS that quietly fixes a very old problem: how do my pods get AWS credentials without me stuffing access keys into Kubernetes secrets?

If you have only ever used IAM roles on EC2 instances, the story so far is: the instance has a role, the instance metadata service hands out temporary credentials, every AWS SDK looks them up automatically. Done. It works because there is exactly one piece of code per instance.

On a Kubernetes node, you have many pods. They have different jobs. The S3-backup pod shouldn’t be able to read the database. The pod that talks to Stripe doesn’t need any AWS access at all. If you hang IAM on the node, every pod on that node gets every permission.

IAM Roles for Service Accounts (IRSA) is the EKS feature that fixes this. A few moving parts; once you see them once they are obvious.

The pieces

The cluster has an OIDC provider — when you create the cluster, EKS gives it a public JWKS endpoint.
You register that OIDC provider as an identity provider in IAM.
You create an IAM role with a trust policy that says “I trust tokens from this OIDC provider, but only if they claim to be service account X in namespace Y.”
You annotate the Kubernetes service account with the role ARN.
EKS’ admission webhook mounts a projected token into pods that use that service account, and sets the right environment variables so the AWS SDK picks it up.

The trust policy

The interesting one is step 3. The trust policy on the role looks like this:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::1111…:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED5…"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED5…:sub": "system:serviceaccount:payments:s3-backup",
        "oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED5…:aud": "sts.amazonaws.com"
      }
    }
  }]
}

The sub condition is the important one — it pins the role to one specific service account in one specific namespace. Without it, anything in the cluster could assume the role.

The service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-backup
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1111…:role/eks-payments-s3-backup

That annotation is all the Kubernetes side needs. The webhook handles the rest at pod-admission time:

Projects a service-account token with audience sts.amazonaws.com into /var/run/secrets/eks.amazonaws.com/serviceaccount/token.
Sets AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE in the pod environment.

The AWS SDK’s default credential chain knows to read that token and call sts:AssumeRoleWithWebIdentity, which gives the pod temporary credentials that rotate themselves.

A Terraform snippet

In practice I would never click any of this in the console. Here is a compact Terraform module sketch that wires the whole thing up. The aws_iam_openid_connect_provider resource is created once per cluster; the role can be created per service account.

data "aws_eks_cluster" "main" { name = var.cluster_name }

data "tls_certificate" "oidc" {
  url = data.aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  url             = data.aws_eks_cluster.main.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.oidc.certificates[0].sha1_fingerprint]
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:${var.namespace}:${var.service_account}"]
    }
  }
}

resource "aws_iam_role" "sa" {
  name               = "eks-${var.namespace}-${var.service_account}"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

Attach whatever managed or inline policies the workload actually needs to that role. The service account in Kubernetes is annotated with aws_iam_role.sa.arn and the rest is plumbing.

The thing people forget

The AWS SDK on the pod also has to be recent enough to support the web-identity credential provider. Anything modern is fine. If you are using a very old SDK or an old version of the CLI that pre-dates IRSA, the SDK will fall back to the node IAM role, you will get inconsistent behavior, and you will lose half a day to it. I have lost the half day, you don’t have to.

Other than that — this is the single piece of EKS that I think every team should set up on day one and never look at again.

Kubernetes in 2026. What changed since the last post here.

2026-01-12T00:00:00Z

The last serious Kubernetes post on this blog was in January 2018 — a CI/CD piece with GitLab and Helm. A lot has changed since. This is not a comprehensive changelog, just the short list of things that changed how the job actually feels.

The control plane is not your problem

In 2018 you set up Kubernetes by running kubeadm on three VMs, hoping the etcd backup script worked, and then writing a runbook for the day a master went down. Today, on every cloud I touch, the control plane is a managed service. EKS, GKE Autopilot, AKS, DOKS, Linode LKE — you ask for a cluster, fifteen minutes later you have a cluster. The vendor handles the masters and etcd.

If you are still running the control plane yourself, you almost certainly have a reason — air-gapped environment, sovereignty, your own metal. If you don’t have a reason, stop doing it.

Helm is not the only answer

I used Helm in the 2018 post and Helm is still around (now on v3, no more Tiller, thank goodness). It is fine. But the alternatives are real now:

Kustomize. Ships with kubectl since 1.14. Overlays instead of templates. If you can read YAML, you can read a Kustomize overlay.
Helmfile / Argo CD applications-of-applications / Flux Kustomizations. Whatever shape you want for managing the chart-of-charts problem.
cdk8s. Define your manifests in TypeScript or Python. Useful if you have repeated patterns that templating handles badly.

For a simple service I default to Kustomize. For something with many parameters and consumers, Helm earns its keep.

GitOps is just how this works now

In 2018 my CI pipeline ran kubectl apply from a runner. This worked until I had three clusters and four people, at which point it started to make me nervous.

The standard pattern now is a tool — Argo CD or Flux — that lives inside the cluster and watches a Git repository. You push a commit, the controller reconciles the cluster to match. The cluster is the only thing with credentials to itself; your CI doesn’t need them.

Side effects of doing this that I didn’t expect in 2018:

The cluster state is auditable from git log.
Rollback is git revert.
The disaster recovery story is “bring up a cluster, point the GitOps controller at the repo, wait.”

Node autoscaling without Cluster Autoscaler

On AWS, Karpenter replaced Cluster Autoscaler in most of my clusters. The model is different: instead of scaling node groups up and down, Karpenter looks at unscheduled pods and provisions exactly the right instance type. If your workload needs 4 vCPUs and 8 GB, you’ll get a c7i.large (or a Spot equivalent), not a node from a pre-baked pool.

The result is fewer empty nodes, faster scale-up, and a much smaller config file. Other clouds have started shipping similar things; on GKE this is roughly what Autopilot does for you.

Ingress is now Gateway API

The Ingress resource is still here and still works, but if you are starting fresh I would use the Gateway API. It separates the roles cleanly: cluster operators manage Gateway resources (think “there is a load balancer here, with this TLS, on this port”), application teams manage HTTPRoute resources (think “my /api path goes to my Service”). This was always how Ingress wanted to be, but you had to express it with annotations.

Observability: OTel everywhere

In 2018 you picked one stack — Prometheus, Datadog, New Relic — and instrumented your code for it. In 2026 you instrument for OpenTelemetry and configure where the data goes. The vendors all accept OTLP. You can keep your instrumentation when you move from one to the other.

The OpenTelemetry Collector is the routing layer. It runs as a DaemonSet, scrapes Prometheus targets, ingests OTLP from your apps, batches, transforms, exports. It is the closest thing to a default I would name today.

A few smaller things

kubectl has built-in JSON Path and --watch and many other things. kubectl get pods -w -o wide covers a lot of needs that used to require third-party tooling.
The k9s TUI is genuinely good. If you live in a terminal, install it.
Secrets in plain YAML are still a bad idea, but ExternalSecrets + a real secret manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) is now an unremarkable setup.
The default container runtime is containerd, not docker. You almost never notice, except when you do.

That is the short list. If your mental model is the 2018 model, you can still navigate; you just look like you’re carrying a flip phone.

Docker multi-stage builds and BuildKit. Shaving off the megabytes.

2025-11-28T00:00:00Z

There’s an earlier post here on Docker OS images from 2017. Most of it still holds. The interesting development since then is that two features — multi-stage builds and BuildKit — make small images much easier than they used to be.

This is a quick walk through both, because Dockerfiles in production still routinely skip them.

The naïve version

Take a Go web server. A first-pass Dockerfile usually looks like this:

FROM golang:1.23
WORKDIR /src
COPY . .
RUN go build -o app ./cmd/server
EXPOSE 8080
CMD ["./app"]

It works. docker build . produces an image. The image is about 950 MB. The actual binary is 14 MB. We are shipping the Go compiler, the standard library source, and a Debian userland to production, every time, because we needed them once during the build.

Multi-stage builds

Multi-stage builds let you define more than one FROM in a Dockerfile. Only the final stage becomes the image. The earlier stages are scratch pads. You can copy artifacts out of them.

# Stage 1: build
FROM golang:1.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/server

# Stage 2: runtime
FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
EXPOSE 8080
ENTRYPOINT ["/app"]

The runtime stage is distroless — no shell, no package manager, just glibc-less static binaries can live there. The final image is around 17 MB. Same binary, same behavior, 55× smaller.

A handful of details that matter:

CGO_ENABLED=0 — without this Go will dynamically link against the build image’s glibc and your binary won’t run in distroless/static. This catches people.
Copy go.mod and go.sum before the rest of the source, then run go mod download. This caches dependencies separately from your code so you don’t re-download them every time you change a Go file.
Pin the base image. golang:1.23 is fine for a personal blog example. For production, pin to a digest.

BuildKit

The classic Docker builder built each stage sequentially, top to bottom. BuildKit is the modern builder that ships with Docker by default since 23.0. It does a few things the old one didn’t:

Parallel stages — independent stages build at the same time.
Mount caches — you can mount a directory across builds for things like /root/.cache/go-build or /var/cache/apt.
Mount secrets — pass a secret into a build without it ending up in a layer.
SSH forwarding for private dependencies, without baking your SSH key into a layer.

The mount cache is the one I would highlight if you have one minute. Here is how it looks for a Node build:

# syntax=docker/dockerfile:1.6
FROM node:20 AS build
WORKDIR /src
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
RUN npm run build

That --mount=type=cache gives the RUN step a persistent cache directory that survives across builds but does not end up in any image layer. On a CI runner this turns a 90-second npm ci into a 5-second one for unchanged dependencies.

The # syntax= line at the top is not a comment — it tells BuildKit which Dockerfile frontend to use. Without it you don’t get the --mount flag and you’ll wonder why your Dockerfile errors out with a syntax error.

A quick checklist

Before you ship a Dockerfile, walk down this list:

Is the runtime image distroless, alpine, or scratch? If it has apt-get in it, why?
Is the dependency install step before the source copy, so it caches independently of your code?
Do you have a .dockerignore? Without one, COPY . . happily copies your node_modules and .git into the build context.
Is the final ENTRYPOINT a non-root user? If not, add USER 1000.

None of this is new. It has all been in the Docker docs since around 2019. It is, however, still missing from most of the Dockerfiles I look at.

A few git commands I wish I knew sooner.

2025-10-14T00:00:00Z

A loose continuation of the Master Git series on this blog — about eight years late, but the topic still comes up.

Three subcommands worth using more often than most people do. None of them are new. All of them are easy to miss in the man pages.

`git bisect` — when did this break?

Imagine the situation. Your test suite was green last month. It is red today. Somewhere between 200 commits ago and now, somebody (very possibly you) introduced the bug. You don’t want to read 200 diffs.

git bisect does a binary search through history for you. You tell it one bad commit and one good commit, and it checks out the midpoint. You run your test. You tell git whether the midpoint was good or bad. It narrows the range. Repeat until you have a single guilty commit.

$ git bisect start
$ git bisect bad           # current HEAD is broken
$ git bisect good v1.4.0   # last release I know was fine
Bisecting: 92 revisions left to test after this (roughly 7 steps)
[a1b2c3d…] some commit message
$ npm test                 # or whatever your check is
$ git bisect bad           # or good
…
$ git bisect reset

For 200 commits you need about 7-8 iterations. If your check is scriptable (and most are), you can let git run it for you with git bisect run <script>:

$ git bisect start HEAD v1.4.0
$ git bisect run ./scripts/repro.sh
…
a1b2c3d is the first bad commit

That is the entire workflow. You start it, you walk away, you come back to a single commit hash. I have used this to track down regressions that I had been staring at for an hour and would have stared at for the rest of the day.

`git worktree` — checking out two branches at once

The standard story: you’re working on a feature branch, your colleague pings you with “quick, can you check what production looks like in main?” You stash. You switch. You forget about the stash for three days.

git worktree lets a single repository have multiple branches checked out into multiple directories at the same time:

$ git worktree add ../site-main main
Preparing worktree (checking out 'main')
HEAD is now at 8f3a2c… some commit

Now you have two directories. Your original one is still on your feature branch with all its untouched changes. ../site-main has main checked out. Same Git history, same remotes, no stash, no tears. When you’re done:

$ git worktree remove ../site-main

I use this for three things mostly:

Reviewing a colleague’s pull request without disturbing my own work.
Running a long-running test on the old branch while I work on the new one.
Comparing builds — I have two different commits compiled at the same time and can diff their outputs.

`git sparse-checkout` — only the bits you need

This one is more situational, but when you need it, you really need it. If you work in a monorepo or any large repository, you can ask Git to only materialize a subset of paths in your working directory. The history is still complete, the objects still get fetched, but your filesystem only has what you asked for.

$ git clone --filter=blob:none --no-checkout git@github.com:org/mono.git
$ cd mono
$ git sparse-checkout init --cone
$ git sparse-checkout set services/payments libs/common
$ git checkout main

The --filter=blob:none bit makes the clone a partial clone (blobs are fetched on demand). --cone mode restricts the patterns to directory prefixes, which is faster and almost always what you want. The result: a repo that pretends, for the purposes of your editor and shell, to only contain services/payments and libs/common.

I will not pretend this is for everyone. If you work on a thirty-thousand-file monorepo, this turns a five-minute clone into a twenty-second one and saves you from having to teach your editor to ignore most of the disk.

That is it

Three commands. None of them were invented in the last year. All of them have been quietly waiting in git --help for me to actually read it.

From Disqus to giscus. Comments without the bloat.

2025-09-02T00:00:00Z

When I put this blog back online, the very first thing I deleted was the Disqus loader on every page.

If you have not looked at a Disqus embed in a while, it has not improved. The script pulls in third-party JavaScript, a small army of trackers, ad iframes for users not paying for Disqus Pro, and a couple of network round trips before it draws anything. For a static site that loads in under 200 ms otherwise, this is just rude.

The cleanup

The original embed lived as a script block on every post. It looked like this, roughly:

<div id="disqus_thread"></div>
<script>
  var disqus_config = function () {
    this.page.url = 'http://artemstar.com/...';
    this.page.identifier = '/...';
  };
  // lazy-load on scroll
</script>
<noscript>Please enable JavaScript to view the comments…</noscript>

One regex pass and 57 of these were gone across 19 files. Removing things is a strangely satisfying step. I wish more of my work was removing things.

What I considered

I had a short list:

Disqus, fixed up. Pay for Pro, remove the ads, accept that you are still loading a third-party SPA. No.
utterances. Open source, GitHub-issue-backed. Lovely, but the project is sleepy and giscus is essentially its successor.
giscus. Same idea as utterances — comments live in GitHub Discussions on a repo you control. Active project, good docs.
Cactus comments / Commento / IsSo. Self-hosted. I do not want a database for this blog.
No comments at all. Honestly tempting. The signal-to-noise on most comment threads is bad. People who want to reach me can email.

I ended up between “no comments” and giscus and decided to leave the blog quiet for now. Giscus is what I would pick the moment I change my mind, so here is the setup that I would use, written down before I forget it.

giscus in five minutes

The flow is:

Pick a public GitHub repo to hold the discussions. A dedicated one is fine, e.g. yourname/blog-comments.
Enable Discussions on that repo.
Install the giscus GitHub App on it.
Go to giscus.app, fill in the repo and mapping options, copy the resulting <script> tag.
Drop the snippet into the comments section of your post template.

The embed looks like this:

<script src="https://giscus.app/client.js"
        data-repo="yourname/blog-comments"
        data-repo-id="R_kg…"
        data-category="General"
        data-category-id="DIC_kw…"
        data-mapping="pathname"
        data-reactions-enabled="1"
        data-emit-metadata="0"
        data-input-position="bottom"
        data-theme="light"
        data-loading="lazy"
        crossorigin="anonymous"
        async>
</script>

The two settings that actually matter are data-mapping="pathname" (so a comment thread is tied to a URL path, not a page title that I might rewrite) and data-loading="lazy" (so the comments don’t load on every page view).

The trade-offs

Things you give up by going from Disqus to giscus:

Anonymous comments. Everyone needs a GitHub account. For a blog about Docker and Terraform that is roughly the same as the audience I had, so I don’t care. For a recipe blog, this would be a terrible choice.
Spam moderation. Mostly not your problem now. It is just GitHub Discussions.
Email notifications. Comes from GitHub. You can subscribe to a thread like any issue.
Comment portability. You own a repository of Markdown discussions. If giscus ever disappears, you still have the data. With Disqus, the data was always theirs.

And the obvious nice thing — the comments box no longer makes its own little Christmas of third-party requests. The blog loads at the same speed whether the section is rendered or not, because nothing happens until the user scrolls to it.

If I end up turning comments on again, the snippet above is what goes in. If not, fewer trackers in the world, mine or anyone else’s.

How to test Terraform built-in functions locally.

2018-03-03T00:00:00Z

Terraform has a bunch of built-in functions that allow to perform common operations when writing infrastructure code. Some of them are so common in many programming languages, that you can guess what they are for even without reading the documentation. For example, you’ll probably recognize the length() function which returns the number of elements in a given list or map, the list() which returns a list consisting of arguments given to a function and the join() which joins a list with the delimiter for a resulting string. You can look through the whole list of these functions in the documenation.

There is a question though. How do you try out these built-in functions locally and see how they work? In this post, I’m going to show a couple of ways you can do that…

Terraform console

Many programming language such as Python and Ruby provide an interactive console as a quick way to try out commands and test pieces of code without creating a file.

Terraform has an interactive console, too. But this console is sort of limited in functionality. It only allows to test interpolations.

Using interpolations you can insert into strings different values which stay unknown before the terraform starts to run. For example, you will often interpolate the value of an input variable into the resource configuration:

resource "aws_s3_bucket" "main" {
  bucket = "${var.bucket_name}"
}

The syntax for interpolation is similar what you can meet in other programming languages. The interpolated value is put inside the curly braces and prefixed with a dollar sign like this ${var.bucket_name}.

terraform console simply creates the interpolation environment. Everything you type into the console is effectively the same as putting it inside ${} in your configuration files. For example, if you launch a terraform console and type a math expression like 1 + 2:

$ terraform console
> 1 + 2
3

You will get 3 in the output. This means that you can wrap the 1 + 2 expression inside ${}, put it in your configuration file and be confident that this will be evaluated to 3. For example, the following will create 3 S3 buckets in AWS Cloud:

resource "aws_s3_bucket" "main" {
  count  = "{1 + 2}"
  bucket = "test-bucket-${count.index}"
}

Now that we understand what functionality terraform console provides, we can use it to test interpolating the built-in functions.

$ terraform console
> list("hello", "world")
[
  hello,
  world
]
> upper("hello world")
HELLO WORLD

But remember, because everything you type will be immediately evaluted, you can’t create variables in interactive console. Thus, if you need to pass a map or a list to your function, you’ll need to create this map or list using map() and list() functions respectively:

> length(list("hello", "world"))
2
> lookup(map("id", "1","message", "hello world" ), "id")
1
> lookup(map("id", "1","message", "hello world" ), "message")
hello world
> lookup(map("id", "1","message", "hello world" ), "author", "None")
None

If you intentionally mistype a function name, you can see the inner workings of terraform console. It wraps everything you type inside ${} and then evaluates the expression:

> lengt(list("hello", "world"))
1:3: unknown function called: lengt in:

${lengt(list("hello", "world"))}  <-- the function gets put inside ${}

Output variables

If you miss creating variables and find working with the interactive console confusing, you can try out the built-in functions using output variables.

Create a configuration file for testing. Then define any variables you want with a variable block and put the expression you want to test inside definition of an output variable.

To demonstrate the same examples with the length() and lookup() function using a new approach, I will create the following test.tf file:

variable "my_list" {
  default = ["hello", "world"]
}

variable "my_map" {
  default = {
    id      = "1"
    message = "hello world"
  }
}

## functions to test
output "my_list_test1" {
  value = "${length(var.my_list)}"
}

output "my_map_test1" {
  value = "${lookup(var.my_map, "id")}"
}

output "my_map_test2" {
  value = "${lookup(var.my_map, "message")}"
}

output "my_map_test3" {
  value = "${lookup(var.my_map, "author", "None")}"
}

If I run terraform apply, I will get the same results as before:

$ terraform init
$ terraform apply
Outputs:

my_list_test1 = 2
my_map_test1 = 1
my_map_test2 = hello world
my_map_test3 = None

P.S. you can find the test.tf file in this GitHub repo.

How to build a CI/CD pipeline using Kubernetes, Gitlab CI, and Helm.

2018-01-15T00:00:00Z

In today’s post I want to share an example of a CI/CD pipeline I created for my test application using very popular nowadays orchestrator Kubernetes (k8s) and Gitlab CI.

Deploy a Kubernetes cluster

NOTE: if you plan to follow my steps make sure to change domain name in the my-cluster/dns.tf config file and make appropriate changes in the name server configuration for your domain.

I’m going to use my terraform-kubernetes repository to quickly deploy a Kubernetes cluster with 3 worker nodes (2 for running my applications and one for Gitlab CI) to Google Cloud Platform.

$ cd ./my-cluster
$ terraform init
$ terraform apply

If terraform ran successfully, you’ll see at the end a gcloud command which you need to run to configure access to the created cluster with kubectl.

$ gcloud container clusters get-credentials my-cluster --zone europe-west1-b --project example-123456

You can check if kubectl is configured correctly by trying to check the server version of k8s:

$ kubectl version

Configure Kubernetes cluster

Next, I will create namespaces for the services I’m going to run in my cluster. I’ll create 2 namespaces for different stages of my example application called raddit and one namespace for running Gitlab CI. All of my cluster specific k8s configuration is contained in my-cluster directory under subdirectory called k8-config:

$ kubectl apply -f ./k8s-config/env-namespaces

I will also create a storage class to use in dynamic volume provisioning for the mongodb service which I’m going to deploy later:

$ kubectl apply -f ./k8s-config/storage-classes/

Deploy Gitlab CI and Kube-Lego

Now, I’m going to deploy kube-lego for automatic handling of Let’s Encrypt SSL certificates:

$ kubectl apply -f ./k8s-config/kube-lego

After that, we’ll deploy Gitlab CI using gitlab-omnibus helm chart with slight customizations made. Make sure to change the IP address to the one from terraform output and use your own domain name:

$ helm init # initialize Helm
$ helm install --name gitlab \
--namespace infra \
--set baseIP=104.155.31.111 \
--set baseDomain=ci.devops-by-practice.fun \
./k8s-config/charts/gitlab-omnibus

It’s going to take about 5-6 minutes for Gitlab CI to come up. You can check the status of the pods with the command below:

$ kubectl get pods -n infra

Create a new group of projects

Once Gitlab CI is up and running, it should be accessible at gitlab.ci.<your-domain> (use root name to log in).

In Gitlab UI, I’ll create a new group for my raddit application:

Gitlab CI groups allow to group related projects. In our example, raddit group will contain the microservices that the raddit application consists of.

Now I will clone a repository with the raddit application:

$ git clone git@github.com:yourname/kubernetes-gitlab-example.git

And create a new project in Gitlab CI web UI for each component of raddit application:

Describe a CI/CD pipeline for each project

Each component of the raddit application is contained in its own repository and has its own CI/CD pipeline defined in a .gitlab-ci-yml file (which has a special meaning for Gitlab CI) stored in the root of each of the component’s directory.

Let’s have a look at the ui service pipeline. Because the pipeline file is long, I’ll break it into pieces and comment on each one of them.

First, we define stages in our pipeline and environment variables. The env vars will be set by Gitlab Runner before running each job:

Although Gitlab CI has its own container registry, in this example, I’m going to use Docker Hub, so if you’re following along, apart from GKE info variables, you’ll need to make sure the CONTAINER_IMAGE variable is set according to your Docker Hub account name.

Also, don’t forget to change the DOMAIN_NAME variable.

Now let’s go to the pipeline’s stages.

In the first (build) stage, we build ui application container, tag it by the name of the branch and commit hash and push it to the Docker Hub registry. Then in the test stage we can run tests for our application which I have none in this example:

In the following release stage, we assign a version tag to the docker images that passed the tests successfully:

The next deploy stage is split into 2 jobs. The differences are very small: I don’t enable ingress for the service running in staging and the deployment to production is manual. The deployment is done using Helm which is a package manager for k8s. You can find helm charts for raddit’s microservices in the root of their directories under the charts subdirectory.

Launch pipeline for each service

The pipeline is already described for each service, but you need to change some env vars as I mentioned above in each of the .gilab-ci.yml files.

To launch a pipeline for a service, we first need to define some secret variables. Click on post project, for example, and go to Settings -> CI/CD. Define CI_REGISTRY_USER and CI_REGISTRY_PASSWORD variables to allow logging to Docker Hub.

Also, define a service_account variable to allow Gitlab to deploy to your k8s cluster. To get a value for a service_account variable just run terraform init and terraform apply in the accounts/service-accounts directory and copy a value from the output.

Define the same variables for ui and mongodb projects.

Now you can push each services folder to the Gitlab repository and the pipeline should start automatically.

Accessing the Raddit application

In the staging namespace you can access the application by using port-forwarding:

and the application deployed into production should be accessible by the domain name via HTTP and HTTPS. Note that it can take up to 5-10 minutes after the first deployment for the application to be reachable by the domain name, because it takes some time to provision a google load balancer:

Destroy the playground

After you’re done playing with Gitlab CI and k8s and wish to delete the GCP resources that you’ve created, run the following commands:

$ helm ls | cut -f1 | tail -n +2 | xargs helm delete --purge # delete all the deployed charts
$ terraform destroy # destroy resources created by terraform

How to visualize your workflow with GitHub projects using AWS Lambda.

2017-08-12T00:00:00Z

The problem

In our company, we use GitHub for source control of our projects. We have tens of different GitHub repos and almost every one of them has outstanding issues and pull requests (PRs) and as the number of projects grows, it becomes very difficult to manage our work on those projects. Although, we receive notifications about new issues and PRs in our chat, they are not organized. We clearly needed a central place to store and visualize all our issues and PRs, so that we could see the problems we have and prioritize our work

GitHub has project boards that allows you to create Kanban boards for your GitHub issues and PRs. But the problem with this is that the process of adding new issues and PRs to a project board is manual. And we clearly didn’t want to go to GitHub and add a new card to the board for every new issue we receive in our chat.

Thus, we decided to automate this process and create a simple GitHub bot using AWS Lambda.

GitHub marketplace

In the GitHub marketplace, I found some project management solutions that could solve our problem. But they literally seemed to cost a fortune.

For example, ZenHub creates a board for each repo with all its issues and PRs. It also provides the functionality of placing multiple repositories’ issues and PRs on a single board. However, because boards are created per repository, to see all your issues and PRs across all of your repositories, you would have to go to some repo and merge all your repos to that repo’s board, which is kind of not very convenient and what we wanted.

Codetree allows your to create a board for opened issues and PRs of multiple repositories and it seemed like it would do for our case, but again the pricing was way too high for our needs.

There is also some other project managements tools like Waffle and Zube, but they all cost too much. I believe those could be handy for teams who have projects under intense development with hundreds or even thousands of issues and PRs. However, we don’t have any such projects, but we do have plenty of projects some of which have issues and it still requires proper visualization and management.

Besides, it seems like GitHub already has the required functionality in place - the project boards. It only required a bit of automation to make the flow of project cards (issues or PRs) across the board visible.

AWS Lambda

AWS Lambda is a computing service provided by Amazon Web Services (AWS) that lets you create serverless applications. With a serverless model, you don’t have to worry about provisioning or managing servers (AWS does it for you), and can simply focus on writing your code. Sounds like a dream for any developer, right? :)

In AWS Lambda, you create a function which will be run in response to certain types of events. It is tightly integrated with other AWS services, so you may choose the events to come from many different sources like S3, CloudWatch, SNS, etc.

In our case, we used GitHub integration with Amazon SNS and then made a Lambda function listen to the topic to which the events were pushed. We’ll talk more about it in a bit.

A great thing about Lambda is that you pay only for the time your code is run and the amount of memory your function uses. Besides, the Lambda includes 1M free requests per month and 400,000 GB-seconds of compute time per month.

With the current rate of invocations of our 128 MB Lambda function, we use the service for free!

Compared to the paid solutions on GitHub marketplace, we save at least 50 dollars a month.

Githubotik

It’s about time we talk about the Lambda function itself.

I wanted to create something very simple. I wrote a python module to interact with the GitHub API the way I needed. The set of functions that this module provides allowed me to perform different actions with the project cards: creating them, moving across the board, deleting them.

Then I started to think about the Lambda function itself. But before I started to write the code, I had to think about what workflow with GitHub issues and PRs we needed. The idea that I had in mind and which we in the end decided to implement was simple:

We have one project board called Backlog for opened issues and PRs across all our repositories. The Backlog has 3 columns: TODO, WIP, DONE. For each opened (or reopened) issue or PR a card is created on the board in the TODO column. If someone from our team takes to work on an issue (or PR if it takes too long to merge), he assigns that issue to himself and a card that represents that issue on the Backlog board is moved to the WIP column which stands for “work in progress”. When an issue gets closed or a PR gets merged/closed, its card is moved to the DONE column. From the DONE column we would remove the cards manually.

As you can see from the description, our project board was going to be a simple Kanban board for visualizing our workflow with GitHub projects and which would be managed mostly automatically by the Lambda function.

Then I created a Lambda function that would implement the described workflow. I will show a piece of that function. The whole version you can see in my repo:

from config.config_loader import ConfigLoader
from github_functions.githubclient import GithubClient
import json


def lambda_handler(event, context):
    config = ConfigLoader().config()
    github = GithubClient(config['org'], config['token'], config['media_type'])

    gitevent = json.loads(event['Records'][0]['Sns']['Message'])

    if gitevent['action'] == "opened":
        if 'pull_request' in gitevent:
            github.add_pull_request_card(
                config['project'], config['column_for_open'],
                gitevent['repository']['name'],
                gitevent['pull_request']['number'], config['labels'])
        else:
            github.add_issue_card(
                config['project'], config['column_for_open'],
                gitevent['repository']['name'], gitevent['issue']['number'],
                gitevent['issue']['id'], config['labels'])

You can see here, we first load the ConfigLoader module. It’s another module that I created to load the configuration settings such as the name of the organization, the name of the project board and its columns, token and media type for talking to GitHub API. This ways all the settings that determine the behavior of the Lambda function are stored in a single json file.

Then in the lambda_handler, which is the actual Lambda function code that will be executed in response to events, we load the configuration, create an instance of GitHubClient class to talk to GitHub API and load the event from SNS topic. Next we look for different fields in the event, which is basically just a JSON object, and if those fields correspond to the activities that we’re waiting for, then we act according to our workflow, i.e. creating or moving the cards across the board.

Example

I’ll show you how you can create a simple Github bot to vizualize your GitHub workflow using code in this repo.

The first thing we’re going to do is to define the variables in the configuration file that we’re loading as part of the Lambda function.

{
  "org":  "githubotik-inc",
  "project": "Backlog",
  "column_for_open": "TODO",
  "column_in_progress": "WIP",
  "column_for_closed": "DONE",
  "token": "XXXXXXXXXXXXXXXXXXXXXXXXX",
  "labels": ["help wanted"],
  "media_type": "application/vnd.github.inertia-preview+json"
}

Most of the settings are pretty much self-explanatory, although I will make a few comments about some of them.

The media type should be passed in the Accept header of all API calls we make, because the Projects API is currently in the preview period. So this setting may need to be changed in the future.

The token is the GitHub token of a user in your organization on which behalf the automatic actions will be performed. In our organization, it’s a special user express42-bot. And in the actual Lambda function we use for ourselves, we don’t specify the token in the config file, but set it as an environment variable and use KMS encryption with our own key, which technically makes the functions cost us 1 dollar a month :)

You can also specify labels that you want to apply to newly made issues and PRs. Although we decided that we didn’t need that and turned it off, the functionality is there so you can turn it on if you like.

Next, we will create the Backlog project board.

Now we can move on to create an SNS topic in AWS Management Console:

Make sure you copy the topic’s ARN, we will need it later.

To allow GitHub to send events to our SNS topic, we need to provide credentials when setting GitHub repo’s integration with Amazon SNS. Therefore, as our next step, we create a new user in IAM:

Make sure your copy the access and secret keys.

As a good security practice, we create a role for our new user giving him only the access he needs. We create a custom inline policy for our new user which will give him rights to publish to the SNS topic we created earlier.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "sns:Publish"
      ],
      "Resource": [
        "<SNS topic ARN>"
      ],
      "Effect": "Allow"
    }
  ]
}

While working with IAM, we also create an AWS Lambda service role for our Lambda function

and attach AWSLambdaBasicExecutionRole managed policy to this role. This will allow our function to write logs to CloudWatch.

finally we’ll give it a name:

Now we add an integration with Amazon SNS to one of our organization repositories. To do that, we’ll use a create_hook script.

Don’t try to add the integration manually, because by default GitHub will only send information about push events to the topic and GitHub UI currently doesn’t allow you to configure it otherwise, so we have to it via API. And anyway, with this script you can add integration to multiple repos with just one command.

But before we can use it, we need to define some variables.

TOKEN = "" # Github token
ORG_NAME = "githubotik-inc" # organization name on Github

## SNS integration configuration
AWS_KEY = ""
AWS_SECRET = ""
SNS_TOPIC = "" # this should be ARN of a topic
SNS_REGION = "" # region where sns topic was created
# see all possible types of events here (https://api.github.com/hooks), look for amazonsns
EVENTS = ["issues", "pull_request"]

We need to provide a GitHub TOKEN to be able talk to GitHub API. As we’re going to change the repository’s settings, the token has to be provided by someone in your team who has access to the those settings.

AWS_KEY,AWS_SECRET,SNS_TOPIC, SNS_REGION variables are values we need to provide to configure a repository integration with Amazon SNS.

EVENTS indicates what type of events we want GitHub to send to SNS topic.

After we defined the variables, we now configure our repository with a simple command like this:

$ python create_hook.py nginx-cookbook

Btw, we can provide multiple arguments to this command to configure multiple repositories.

You can now open a repository’s page in your browser to make sure it’s configured correctly.

We finally come to the last step which is creating a Lambda function in AWS :)

In order to create a Lambda function in AWS, we need to provide a zip archive of our Lambda function code, as well as all its dependencies.

In this step, we could use some advanced ways to create a Lambda function. For example, the Serverless framework looks pretty cool. However, in our simple case, I think it would be unnecessary, as it requires time to install the framework on your machine and learn it. So I decided to go with a good old Makefile :)

In fact, I took the idea with the Makefile from this video and it seemed like it would fit perfectly for our project.

When you’re done with the code, simply run two commands to package your code and dependencies:

$ make  # create a virtual env
$ make build  # package a function in zip format

The zip archive will be created in the following path package/githubotik.zip.

Let’s go now to AWS Management Console once again and create a Lambda function.

Go to AWS Lambda service page and look for sns blueprint.

Add sns topic we created earlier as the trigger and enable it:

Set the name of the function and choose the runtime environment:

Choose to upload a zip package and upload our function:

Fill in the Handler field according to the instructions and select the role for our function:

Click advanced settings to configure the memory usage and timeout. For our simple function, 128 MB of memory will be more than enough and it hardly ever takes to run longer than 5 seconds.

That’s it. Now let’s try it out! In the video below, I’ll demonstrate how the work with issues and PRs is now visualized on the project board:

Conclusion

Another problem that we faced when we started to use this Github bot was that we had to put all our existing issues and PRs on the board. If you decide to use it, you may find useful this script which does it automatically for you.

We’ve been using this bot for over a month now and so far it really made our work with Github projects visible and more manageable.

Master Linux CLI (part I). Useful Linux commands.

2017-08-03T00:00:00Z

Interestingly, when you start reading books about Linux or going through different tutorials, they often don’t tell you about some of the cool commands that make your work sometimes so much easier. Maybe they are hiding them from you to make sure you still have to learn something in the future? ¯\_(ツ)_/¯

Anyway, today I decided to make a quick overview of some of the commands which I find useful, but which are sometimes hard to find out about.

tee

tee command allows you to write to the stdout and a file (or files) at the same time.

This is useful when you want to store and view the output of any command.

And you can also use it to save the output to multiple files.

This command is incredibly useful when you want to store the output of a command to a file but also redirect it as an input to another command.

As you can see, we are able to take snapshots of the data as it flows through the pipes.

pbcopy (Mac) or xclip (Linux)

This allows you to copy a file’s content to the clipboard.

Now if you try to paste, you’ll get cucaracha, which by the way means a cockroach in Spanish :)

This command makes copying from the terminal a breeze. I find it especially useful when I need to copy SSH or GPG keys.

watch

watch runs a specified command repeatedly at regular intervals and displays its output on a console.

This is used when you need to continuously monitor some command’s output.

Simple examples include monitoring who is logged in to the system with watch who command or watching for changes inside a directory with watch ls

We should note the -d option that allows you to highlight the changes that happen in the command’s output.

Just to show you how it works, we’ll use it with a date command.

script & scriptreplay

These are 2 awesome commands which allow you to record and replay a shell session for you.

To use a script command, we can just type script in which case the session will be stored in a default file named typescript. We can also specify the name of the file in which we want to store our session as the first argument to the script command.

An alternative to the script command is history, but it only keeps track of the commands you use and not their outputs.

Another cool thing about the script command is that the shell session that you have recorded can then be replayed in your terminal with a scriptreplay command. This is particularly helpful if during the session you start interacting with some programs like htop.

To be able to replay a recorded shell session, we need to specify a filename for storing the timing information.

$ script --timing=time.txt myshell.log

Then after we’re done recording, we can use the scriptreplaycommand to replay the session.

$ scriptreplay --timing=time.txt myshell.info

It’s also important to note the -c option to this command which allows to record the output of a single command. For example, this might come in handy when we need to record a command’s output in our bash script. The syntax goes like this:

$ script -c 'ping -c 3 google.com' myshell3.log

jq

jq is a handy JSON processor that allows you to extract necessary fields from a JSON object.

Let’s take a simple JSON file and extract some specific fields.

{
  "colors":
    {
      "color": "black",
      "category": "hue",
      "type": "primary",
      "code": {
        "rgba": [255,255,255,1],
        "hex": "#000"
      }
    }
}

env

If being run without any arguments, this command will show you a list of current environment variables.

It also allows you to run commands with specific environment variables without actually changing your environment.

This can be helpful when you need to run a one time command that requires some specific env variable, but you don’t really want to change your environment. For example, to build a GO binary for Ubuntu on my Macbook I would run a command like this.

$ env GOOS=linux GOARCH=amd64 go build src/hello-world.go

This runs the go build command with two additional environment variables. If I run env command right after that, I won’t find those variables in my environment.

Hopefully, you’ve found this post useful. And if you have some interesting commands to share with me, please leave a comment below!

What are Docker OS images and why would I want to use them in my Dockerfile?

2017-07-23T00:00:00Z

Docker is not new these days. Everybody knows how to run a docker container at least on a local machine, because it’s so easy. You find an image on DockerHub, you run docker run -d <image-name> and that’s it.

You probably also know how to build docker images, because all you need is create a Dockerfile, use 7-8 commands to describe how to package your application and all its dependencies, finally build the container image with docker build command and run it.

Let’s look at this simple Dockerfile.

FROM ubuntu:14.04
COPY ./hello-world .
EXPOSE 8080
CMD [ "./hello-world" ]

It looks simple, right? Nothing special.

You specify the OS image, add you binary file, … wait, it just struck me. Look again, at the Dockerfile.

What in the world the ubuntu OS does here? If a container needs an operating system, then how is it different from a virtual machine?

A virtual machine has an operating system and runs our applications inside. So how different is a docker container then? Is it some type of virtual machine?

Was all that talk about a container represents a bundle of your application and its dependencies just a big lie? (☉_☉)

To really understand the difference between the VMs and containers I offer you to look at a bit of history. I want to specifically look at how these technologies came into play and what problems they were meant to solve.

Let’s first start by looking at how things were before people started using VMs. The traditional server model that was used for decades in IT looked like this.

Here we have some vendor hardware and an operating system which runs on top of it. The operating system is bound to the hardware by the drivers which are specific to that hardware. Finally, on top of the OS we run various types of applications and services.

This model has lots of limitations.

First, it leads to lots of underutilized hardware. In most cases, the applications and services that you run on your server don’t utilize even half of its computing resources. In fact, previous studies showed that the average utilization of the server’s resources was about 5-10%. Which means that if you spend 10 thousand dollars on a server hardware, you waste 9 thousand dollars worth of computing power. Plus, you have to spend a lot of money on the servers maintenance.

But why do we don’t utilize all of our computing resources? Why can’t we just run more applications on our servers? (・ω・)b

Well, it’s not that simple as it might seem. There are some challenges that we face at OS level which include running multiple instances of an application or running different versions of the same application at the same time.

Most of the applications you see have some dependencies that need to be provided in order for them to run. And the problem we often see is that two applications may depend on the same library but require different versions of that library. For example, this might be the case when we want to run different versions of the same application. Installing two versions of the same library system-wide and telling each of our applications which one they need to use would be a mess and in most cases is not even possible.

We can see how this problem is addressed in some programming languages. For instance, in python, you use virtual environments to create a separate isolated python runtime environment with all the dependencies specific to each application. This way we can run different python applications without breaking each applications’ dependencies.

Although, the problem with application’s dependencies is solved in some programming languages, it’s not solved for all applications that we use. We need to have some technology which would allow us to provide isolation of runtime environments for any application.

Besides, what if we decide to increase our computing resource utilization by running multiple instances of the same application on one server?

We find many problems here as well. For example, running multiple instances of nginx would require changing its init script, as well as its configs because we can’t bind more than one application to the same port. As you can see, it takes quite a bit of work.

I think you get the idea that we have quite a few things to think about at the OS level of our server model. What do we do? 🤔

A bunch of people came up with this idea that if we can’t run multiple applications in one OS, let’s just figure out a way to run more OSes on our physical servers. Thus, the hardware virtualization came into play.

The idea was that a special piece of software called Hypervisor abstracts the underlying hardware (virtualizes it) to provide the same hardware to several operating systems. Operating systems which run on top of Hypervisor talk to the virtual hardware presented by the Hypervisor using the generic drivers. Thus, we could have multiple OSes of different sorts running on the same server at the same time each using a piece of computing resources that are there.

Thanks to hardware virtualization, the OS became portable. We’d seen before how the operating system was bound to the vendor hardware. But now operating systems can’t see the server’s underlying hardware, instead they see the generic drivers and the generic hardware provided by the Hypervisor. This fact makes the process of distributing and running various OSes on any hardware very easy. All we need is a proper hypervisor running on that hardware or host OS.

Hardware virtualization allowed us to run in isolation multiple OSes on the same physical server and made the movement of an OS form one machine to another much easier.

Did it solve our problem with computing resource utilization? It sort of did. We can now run more instances our applications on a physical server by running more instances of operating systems.

But it doesn’t really solve our initial problem with running multiple applications on the same OS. It seems more like a workaround.

What we really need is a technology that would allow us to run our applications in isolation on one OS.

And this is where containers come in.

Containers are an abstraction at the OS layer. With the help of containers, we can bundle the application code and all its dependencies into one standardized package and then run it in isolation from other system processes.

Isolation means everything here. For instance, filesystem isolation means that each container gets its own filesystem which solves the problem with version conflicts. Network isolation means each instance gets its own IP address and thus two containerized applications are free to bind to the same port, without us having to deal with any init scripts.

This really solves our problem with resource utilization. We can now run multiple instances of applications (even of the same type) in our OS and use our computing resources to the maximum.

Since container technology is based on the features built into the Linux kernel, like cgroups and namespaces, we can only run containers on Linux distributions. Although, with the use of virtual machines, we can run containers on pretty much any OS (check how Docker works on Windows or OS X).

The Docker itself is a software container platform which allows us to create and manage containers.

Docker provides a standard format for creating container images. This makes it extremely easy to run and distribute our applications through various OSes. For instance, we can now build a container image with our application on CentOS and then run that same image on Ubuntu without having to change anything, because differences in OS distributions are abstracted away. The only thing we need is to have Docker installed.

But I want to note that having containers doesn’t mean we don’t VMs. They are still widely used and help us decrease hardware costs. They are often used in cases when we need to provide computing resources to different people. Virtual machines allows us to allocate the exact number of computing resources to meet the client’s requirements, and if there’s some resources left we can allocate it to another client. This is how public clouds work.

In fact, VMs and container nicely work together (＾＾)ｂ

As for using VMs to provide isolation for our applications, VMs seem a bit too heavy. They are big in size (we’re usually talking about gigabytes here) because they run the full OS which includes things like the kernel and hardware drivers.

On the other hand, containers don’t need all that software the OS usually have. All they need is an application you want to run plus its dependencies. As a result, containers are much smaller in size (MBs against GBs) which means they’re easier to build and distribute. They are also start and stop quicker because there’s no OS boot process required.

So containers are obviously a better choice for running applications in isolation. Although, VMs are still used for running and distributing applications when a higher level of security and isolation is required.

This is all great, but I still don’t get it why we have the ubuntu image in the Dockerifle? If a container provides isolation to the application and its dependencies, and uses the host operating system’s kernel, then why do we need to specify an OS image in our Dockerfile (◔_◔)???

Let’s make things even more confusing before we answer that question.

If I build the image from the Dockerfile above and look at its size, here’s what I will see.

Ubuntu image is only 188MB. 🤔

Have you seen ubuntu OS image of that size? If you try to search for ubuntu 14.04, you’ll see that its size is about 1GB.

So why is ubuntu container is 5 times smaller than a normal ubuntu image?

As we talked earlier, containers don’t need an OS as they share the kernel and execute instructions on the host directly. So that ubuntu container images is stripped down image of a real operating system which doesn’t include kernel, but does have some libs and utilities specific to Ubuntu distro.

This means that we actually don’t need ubuntu container image to run my GO binary. If I try to build another container image (v2.0) for my test application from scratch, it should still work.

FROM scratch
COPY ./hello-world .
EXPOSE 8080
CMD [ "./hello-world" ]

scratch is a reserved docker image name which just means no-op.

Noticed how it skipped the first step and went straight to the next?

Now if I look at the image size of my new image, I’ll see that it became significantly smaller - only 5 MB against 194 MB! And it still works as before.

But why people still use OS container images anyway? One of the most common reasons for this is a package manager. When you write a Dockerfile to build your image, it’s much easier to install required dependencies inside the container using a package manager in the RUN command than directly copying all the packages from your local machine to the container.

Another reason to use an OS image is it facilitates troubleshooting. For instance, OS image may have a shell installed, this way I can run a virtual terminal inside the container and then issue commands from inside.

Let’s see if can I check internet connectivity from inside the containers that I’ve built.

As you saw, I wasn’t able to run a shell inside the version 2 of my container which was built from scratch. But I could run shell and ping inside the version 1 of my container, because the parent container (ubuntu) has all the required binaries in it.

So using OS images when building your application container images is a common practice. But you should remember to choose the OS container image wisely to avoid bloating your application container image.

For further reading, I recommend checking out the Alpine Docker image which is one of the most popular these days. It’s very lightweight (4-5 MB), it has a package manager and many useful utilities preinstalled.

Iterm2 + Tmux = Awesome

2017-05-28T00:00:00Z

If you regularly work with the terminal, especially if your work with remote servers via SSH, you must know about Tmux. It is an awesome tool that makes your work so much easier!

What is Tmux? Tmux stands for a terminal multiplexer. It basically allows you to open multiple terminal sessions inside a single terminal window or even remote terminal session (like when you SSH into a server). This may not seem like very cool to you now, but let’s look at some examples of how it works and I’m sure you’ll love it :)

First, we need to install and configure Tmux. As I also wanted to show you how awesome Tmux integrates with Iterm2, I assume you have it ╭( ･ㅂ･)و

To install Tmux just run:

$ brew install tmux

Then type tmux in your terminal, hit enter and start using it ;)

Well, you’ll probably get lost if you use it for the first time as it takes a few things to learn to get a grasp of it. So follow along and I’ll show you how things work.

As we already mentioned, Tmux allows you to open multiple terminals inside a single terminal window. But it also supports splitting panes which allows you to open multiple terminals inside a single terminal window so it looks something like this:

Tmux is developed as a client-server model which brings the concept of sessions into play. You create those new terminal windows within a session. So the first thing we need to know is how to create a session:

$ tmux # creates a new session (session number is assigned as a session's name)
$ tmux new -s <session-name> # creates a new session and assigns it a name

To list the sessions you can run:

$ tmux ls

Creating a session automatically creates a new terminal window within that session. You’ll see how your current terminal window is switched to a new terminal.

You may wonder how did I detach from the new session and get to my original terminal. Here is the command:

Crtl + b, d # don't type the comma, it is here to separate the prefix and the command

Here Crlt + b is the so called prefix which basically tells the terminal that the next thing you type will be a Tmux command. You’ll have to deal with this prefix a lot if you don’t use Iterm2, as every Tmux command should be prefixed with this combination, although you can change it.

To attach to an existing session you can use the following command:

$ tmux a -t <session-name> # "a" is short for "attach" here

Now as you got the basics, let’s look at a more advanced example with splitting panes.

“That’s all very cool, but I can do the same things with Iterm2 profiles and splitting panes” - you’ll probably say. And you’re right. I personally very rarely use Tmux locally. When Tmux becomes incredibly useful is when you work with remote servers via SSH.

Does it ever happen when you need to work with some remote server regularly via SSH? Maybe you’re a web developer and develop your new application. Or maybe you’re a system administator who is testing a new service setup and configuration. In these cases, you’ll often open up many terminals: one - for a text editor with a service configuration or code, one - for tailing logs, one for launching the application or service, etc. Your work on a server can take hours and what really sucks is when you take a break and go for a lunch and you need to keep your ssh connection open, because otherwise you’ll have to open all those windows again to be able to work.

Here is where Tmux comes into play. The greatest thing about Tmux session is that it can persist beyond SSH logout. So you can just shutdown your laptop, go play soccer, then come back, ssh into your remote server and open the terminal windows as they were when you logged out. Tmux basically makes your work independent of SSH connection.

Let’s see an example that demonstrates that Tmux sessions indeed persist beyond SSH logout.

Here I used this cool Iterm2 and Tmux integration I was talking about.

Instead of switching my terminal to a new one created by Tmux session, Iterm2 allows me to open it just like another tab in my terminal window. Moreover, I don’t have to memorize all of those Tmux commands like Crtl + b, % to split a pane, Crtl + b, o to switch a pane, but instead I can use the Iterm2 shortcuts I use every day \ (•◡•) /

To use Tmux with Iterm2, you only need to provide an extra option (-CC) for Tmux commands:

$ tmux -CC # create a new session
$ tmux -CC a -t <session-name> # attach to a session

And yeah, you can choose if you want to open Tmux session in a new tab or a new window. See Iterm2 settings:

Tmux is very lightweight and even comes preinstalled with some Linux distributions like Ubuntu. Tmux eliminates the risks of losing an SSH connection. Sometimes it’s very important like when you’re doing a manual backup of your server. In gereral, if losing SSH connection means a lot of work to you - opening a window for a text editor, for logs, for running commands - then Tmux can make your life so much easier. Let’s look at another example close to a real world use case: playing around with nginx configuration:

Hopefully, if this post didn’t make you fall in love with Tmux, at least it made you curious ⚆ _ ⚆

In the end, I want to also mention another cool feature which Tmux provides. It allows multiple users to share a terminal session which basically means two people can work in the same terminal at the same time. For example, this could be helpful for pair programming.

Also, if you really liked the idea behind Tmux, you may want to take a look at Tmuxinator which allows you to customize your work with Tmux even further.

AWS Roles. When and how I can use them?

2017-05-17T00:00:00Z

Unless you work via AWS Management Console, in order to access AWS resources you talk to AWS API. All API requests that you make need to be signed by secret access keys (access key ID and secret access key), so that AWS could identify who is making the request and prevent strangers from accessing your resources.

If you ever worked with tools like AWS CLI and Terraform which allow you to manage your AWS infrastructure, then you know that they require AWS credentials in order to work, because they work by making API requests to AWS API. When you work with these tools, you usually put your credentials to a special .aws folder under your user, and they simply take it from there every time you use them.

But what do you do when you need to provide access to AWS resources to some scripts or an application running on your EC2 instance? For example, your application running on an EC2 instance might need to save or fetch some files from an S3 bucket. Do you just put user credentials on that instance in this case? Although this will work, there’s a better way to do this…

For cases like this, when one AWS service needs to access another, AWS offers a special IAM entity that is called a role.

Roles are similar in concept to users. It’s an identity to which you attach policies with permissions to access AWS resources. The difference is that, unlike a user, role is not associated with a single person, instead it is meant to be assumable by any person or any service that needs it.

What roles bring to the table:

Dynamic credentials. Unlike users, roles don’t have any credentials associated with them. Instead, credentials are created dynamically and provided to a user or a service that assumes the role. This saves us some trouble of having to distribute credentials.
Temporary credentials. Each set of credentials that you use requires a rotation. Great thing about roles is that credentials that come with it are rotated regularly for you. If you chose to put user credentials on your machines, the rotation would mean creating a new user account and running configuration management tools to update credentials on all of your servers. This would mean extra work.

To sum it up, EC2 instances that have an IAM role attached automatically have AWS security credentials available which are also regularly rotated for you. Thus, your application or any script that you run on that instance can use those credentials to access AWS resources via API. And you, the person who manages the AWS infrastructure, are saved quite a bit of time and trouble related to security credentials rotation and distribution. Sounds cool? Let’s see how it works.

Example

We will first create an IAM role and attach a policy that gives access to an S3 bucket. Then we launch an EC2 instance with this role attached, ssh into the instance, and do some tests like uploading and downloading files from S3.

When creating a role via Amazon Management Console, it does some steps for you automatically, that’s why to show you how this process really goes I will use AWS CLI.

1. We’ll start by creating a file with our trust policy. This is one of the two types of policies we’ll need to create. A trust policy simply describes who can assume this role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}

In the principle element we specify a user, AWS account, AWS service, or other principal entity that is allowed or denied access to a resource. In this trust policy, we say that we want to allow EC2 service (our instances) to assume the role which in turn means retrieving temporary credentials from AWS STS.

2. After we’ve created a file with our trust policy, we’ll create the role itself.

$ aws iam create-role --role-name logsbucket-role --assume-role-policy-document file://trust-policy.json

I provide the path to a file with the trust policy after the file:// prefix.

3. Now that we’ve created a role and defined who can assume it (the principle), we need to add a set of permissions to our role. We’ll create a permissions policy which defines what actions and resources the principal is allowed to use.

First, I’ll create a file (logs-bucket-permissions.json) with my access policy:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:PutObject",
              "s3:GetObject"
          ],
          "Resource": [
              "arn:aws:s3:::artem-server-logs/*"
          ]
      }
  ]
}

Note, that each resource in AWS is uniquely identified by the Amazon Resource Name (ARN), so we need to specify ARNs of those resources we want to make accessible through this role.

In this policy, we allow to get objects from and put object to my test bucket artem-server-logs which I created beforehand.

4. Then we have two options: either to embed this policy in our role making it inline policy (put-role-policy command) or create a standalone (managed) policy and then attach it to the role (attach-role-policy command). The difference between managed and inline policies is explained in full detail here.

In this example, we’ll use inline policy. This way it will be deleted along with the role when I delete the role.

$ aws iam put-role-policy --role-name logsbucket-role --policy-name LogsBucketPermissions  --policy-document file://logs-bucket-permissions.json

5. We’ve created our role and given it permissions, but there is still one thing we need to do before we can use it with EC2 instances. The problem is that we cannot directly attach the role to an EC2 instance, because the application that is running on it is abstracted from AWS by the virtualized operating system (read more on this here). To assign a role to an instance and make it available for the application or script that is running on it, we need to put the role in a container that is called instance profile.

I took a picture from this post where they did a good job explaining how roles work with EC2 instance.

This picture shows you that instance profile is just a container that you attach to an EC2 instance and that holds your role. An application running on the instance has access to the role through this container and uses the role to retrieve temporary credentials for signing API requests.

When you create a role via Amazon Management Console, the instance profile is created for you automatically. It has the same name as the role. But it doesn’t work this way with AWS CLI, so we need to create a container ourselves:

$ aws iam create-instance-profile --instance-profile-name logsbucket-profile

and then put our role in this container:

$ aws iam add-role-to-instance-profile --instance-profile-name logsbucket-profile --role-name logsbucket-role

You can see what role the instance-profile contains by running:

$ aws iam get-instance-profile --instance-profile-name logsbucket-profile

Things to note: instance profile can contain only one role, but a single role can be included in multiple instance profiles.

6. We’re done with the role now. The only thing left is to lauch an EC2 instance with our role and test it :)

I will launch Ubuntu 16.04 instance via Managements Console since AWS CLI command seems to be too long to paste it here. In the third step of the launch wizard, you can specify which role to attach to the instance.

As you can see from the picture, instead of roles you actually choose an instance profile that holds your role.

7. After instance has been launched, I ssh into it and check if I can retrieve temporary security credentials that come with the role.

The security credentials are retrieved from the instance metadata. The instance metadata is basically all the data about the instance that is accessible from within the instance itself. So if an application needs to know some information about the instance it’s running on, including temporary security credentials which are available to the instance, it needs to send an HTTP GET requests to the following URL:

$ curl http://169.254.169.254/latest/meta-data/<metadata-category>

Thus, if I want to test whether the security credentials are available to me from within the instance, I can try to get them from the instance metadata:

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/logsbucket-role

Most of the times you will use AWS SDK for making AWS API requests. In this case, you don’t have to make requests to the instance metadata to retrieve AWS credentials, AWS SDK does this for you.

AWS CLI is a command line tool built on top of the AWS SDK for Python. It’s used often to manage AWS resources via API. We’ll use it for our tests.

I’ve just tested if I could retrieve security credentials from metadata, now I want to check if I can actually get files from and save files to my test S3 bucket.

I will install AWS CLI on my ubuntu instance:

$ sudo apt-get -y update
$ sudo apt-get -y install awscli

Then I do my testing. I create a file, upload it to my test bucket, delete my local copy and download it again from the bucket.

It works as expected, but you might notice that I also had to specify the region of my bucket through the environment variable.

Conclusions

In this post we’ve seen how AWS Roles work and what benefits we get from using them.

AWS Roles are used all the time in many different cases.We tried uploading a test text file to an S3 bucket and downloading it from it, but the same way we could archive and upload system and application logs to a remote storage. The application that you run on your EC2 instance might need to access DynamoDB (NoSQL database) which is another AWS service and we see the same problem with credentials management which we now know is easily solved by using roles. Thus, it’s really important to understand how AWS roles work, so that we’re not afraid of using them :)

Here is another interesting link to the video that shows you how AWS roles could be used with SDK-based applications.

Master Git (part V). Change commits in your history. Interactive rebase.

2017-05-07T00:00:00Z

When you work with with version control, you sometimes find yourself in situations when you would like to edit or delete the saved changes in your project’s history. Well, in this case, you should know that rewriting your project’s history could be dangerous.

Unless you work on a project alone, you will have to share your changes to the project (commits) with other people. In most cases, you share commits via a central server like Bitbucket or Github other than directly with other developers. I’m sure you already know that when we make a new commit it’s based on some other commit(s) which is considered to be the new commit’s parent(s). So once you shared your commits, they will constitute the basis for the work of other people. Rewriting or deleting commits that you already shared with others would mean a whole lot of trouble for those who based their work on those commits. So unless you want your colleagues to hate you, follow this simple rule:

Never change commits which you already shared with others.

If you want to undo the changes in your project’s history, just create a new commit that corrects or completely removes the changes introduced by previous commits. There is the revert command which allows you to do just that.

If you understood that rewriting your public history is bad, then you’re safe to keep reading :) Today I will show you how changing commits could be useful.

I already talked about commands like git commit –amend which helps you to fix up the last commit and git reset that allows you to restore to the previous commit removing all the commits following it.

Again, those are very useful, but should be used on local commits only.

Today we will talk about a more powerful command, that is git rebase -i, which gives you full control on how you want to reshape your project’s history.

Rebase

Remember what rebasing is? It simply allows you to change the base of your current branch. Basically this means taking the changes introduced by commits and creating new commits with the same changes but with a different base.

-i option starts interactive mode which gives you full control on how this process goes.

Case 1: Two different branches

So let’s imagine we have the same situation as depicted on the picture

and from the feature branch let’s run interactive rebase.

$ git checkout feature
$ git rebase -i master

This will open up a text editor and offer you different options on how to create new commits. In the comments, described all the things we can do.

As we can see, we can edit existing commits (edit) including their messages (reword), merge several commits into one (squash, fixup), run commands as we create new commits (exec), and remove commits (drop).

Let’s imagine that my two commits added new functions which are closely related and work in couple. In this case, I may want to squash the second commit into the first and leave in project’s history a single commit instead of two. I will use the squash option which, unlike fixup, allows me to change the message for the resulting commit.

So I change pick to s(squash) for the second commit.

After saving and exiting, it will open up a text editor again, asking me to edit the commit message. It shows all the messages for the commits. If we leave it as is, all of them will be used for the the new commit’s message.

So I’ll just edit the first one and delete the rest which in my case is just the second commit message:

In the end, we get one new commit which applies to a new base (master) the same changes as the two commits we just squashed.

Case 2: the same branch

Interactive rebase can be useful not only when we rebase one branch onto another. It is often used to rewrite the history of a branch - the thing we talked about in the beginning when we mentioned commands like amend and reset.

Let’s imagine I have a local branch with a series of commits which I haven’t shared with anyone yet. And before I do share it with anyone, I would like to edit some of them, probably squash a few as I made a bunch of small commits which better look together, maybe correct some typos in the commit messages, etc.

First, I choose the range of commits which I want to change. I will change commits that follow the commit bdc72fe - this will be the base for all new commits which will be created - remember rebasing creates new commits at the specified base.

$ git rebase -i bdc72fe

This will open up a text editor as we’ve seen before and list the commits coming after commit we specified which is bdc72fe.

My commits regarding the timeout served the same purpose of introducing a new parameter to the function and removing a temporary workaround. So I’ll squash (s) all those commits in one. I also noticed that I made a typo in the word controller, so I’ll reword that message (r). And the last commit introduced a new function which I decided to be unnecessary, so I’ll drop that commit (d).

By changing commit messages as described earlier, I achieve the state of the project’s history I want:

How to undo rebase O_o ?

Git allows you to restore history even when you rewrite it. I have already touched on using reflog to restore changed history to the previous state in one of my posts.

So I’ll simply show you how I restore my project’s history to the state before we started rebasing.

We use command git reflog to find the commit to which HEAD pointed to before we started the rebase. It’s usually the last commit that you’ve made.

the I’ll use hard reset in order to restore to the state when the last commit was added to my branch and before we started rebasing.

$ git reset --hard HEAD@{17}

How to work with multiple AWS accounts.

2017-04-22T00:00:00Z

Ever had to work with multiple AWS accounts? If so, then you probably have a working solution on how to make switching accounts easier, in which case don’t hesitate to share it with me in the comments below. But if you still find troublesome managing your multiple AWS credentials, then you should find this post interesting.

If you’re used to working with just one AWS account, you most probably used aws configure command to quickly set up your default credentials like this.

Recently, I find myself in a situation, when I had to switch between two AWS accounts of two different companies. As it’s clearly not a rare case, the first thing I did was to look for a solution on how to manage multiple profiles that Amazon itself would suggest.

And I found that Amazon allows you to create named profiles for each set of your credentials.

So I created two profiles for each AWS account that I had to work with. I chose to name those profile according to the names of the organizations to which those user accounts belonged.

I forgot to mention that I deleted the default profile which I had previously configured. I did this so that my actions wouldn’t affect any AWS resources without me being fully aware of where I’m going to do the changes.

Now, I could use AWS CLI with 2 different user accounts as long as I provided --profile <profile_name> option to each command:

aws ec2 describe-instances --profile ex

By the way, this command to describe instances gives too big of an output. So I usually use the following alias to get a short description of launched instances:

# ~/.zshrc
alias idesc="aws ec2 describe-instances --query 'Reservations[*].Instances[*].[Placement.AvailabilityZone, State.Name, InstanceId,InstanceType,Tags]' --output text"

But having always to provide the profile option in every command can make you quickly tired, right? Good thing, Amazon allows us to use an environment variable to specify the profile we want to use:

Setting the AWS_PROFILE environment variable affects credential loading for all officially supported AWS SDKs and Tools (including the AWS CLI and Terraform).

Now new questions arise. The environment variable for a profile is great, but where we define it and how we get information about which profile we’re using.

At first, I looked for solutions on the internet, but I didn’t find any to my liking. So I came up with a simple bash script which would provide me with ability to quickly switch profiles, turn them off, and give me the visibility into what profile I’m currently using.

Here is how my script looks like:

if [[ $1 = 'on' ]]; then
  if ! aws configure --profile $2 list &> /dev/null ; then
    echo "profile \"$2\" doesn't exist"
  else
    if ! grep "export PS1" ~/.zshrc &> /dev/null ; then
      echo "export AWS_PROFILE=$2" >> ~/.zshrc
      echo "export PS1=\"($2)\$PS1\"" >> ~/.zshrc
    else
      sed -i -e "s/.*export PS1.*/export PS1=\"($2)\$PS1\"/" ~/.zshrc
      sed -i -e "s/.*export AWS_PROFILE.*/export AWS_PROFILE=$2/" ~/.zshrc
    fi
    source ~/.zshrc
  fi

elif [[ $1 = 'off' ]]; then
  sed -i -e '/.*export AWS_PROFILE.*/d' ~/.zshrc
  sed -i -e '/.*export PS1=\(.*\).*/d' ~/.zshrc
  source ~/.zshrc
  unset AWS_PROFILE
else
  echo "Usage:"
  echo "To switch to a specific profile: awspr on profile-name"
  echo "To turn this thing off: awspr off"
fi

As you can see, I export AWS_PROFILE in my ~/.zshrc file, so that when I choose to switch to a specific profile I could open up new panes in my terminal or even multiple terminal windows and still work the same AWS profile.

I also change PS1 variable which defines how my command prompt will look like. I add the name of the profile to which I switched at the very beginning of my prompt. This way I can always see what profile I’m using at this moment.

I placed this script under ~/bin folder (~/bin/awspr.sh) and made it executable. Another thing that I did to start using this script was to create a new alias in ~/.zshrc:

alias awspr=". ~/bin/awspr.sh"

This launches my script in the current shell when I run awspr command.

That’s it. Now, to switch to a specific profile I run awspr on <profile_name>. And if I don’t work with AWS and don’t want to see a profile name in the command prompt, I can turn this thing off by running awspr off:

P.S. the script that I posted here could be easily customized to work with other shells and different linux distributions. My goal was to write something quickly for my personal use.

Master Git (part IV). Stash your changes

2017-03-29T00:00:00Z

Imagine a situation when you start working on some part of your project, make a bunch of uncommitted changes, but something urgent comes up that requires you to quickly make a few commits concerning another part of the project. In such cases, instead of losing the work you have already done, you can use git stash command to save your uncommitted changes away for later use while switching to another task.

Stashing in git is simple. All you need to do is to run

$ git stash

and git will make it look like those uncommitted changes are gone and your repository is clean:

And as you see, to recover stashed changes you use command

$ git stash pop

The thing you should be aware of is that git doesn’t stash untracked files:

If we want to stash untracked file, we need to pass -u option:

You can stash more than once. In this case, you probably want to add messages to your stashes with

$ git stash save "message"

To see a list of all your stashes:

$ git stash list

You manage your stashes by first looking at their identifier with git stash list command and then using commands:

$ git pop <stash_id> # to re-apply a stash
$ git drop <stash_id> # to delete a stash
$ git stash clear # delete all stashes

Master Git (part III). Restore undone commits.

2017-03-28T00:00:00Z

In cases when you use git reset --hard to undo some commits, you basically erase commits. In cases you happen to change your mind about the commits you deleted, Git still provides an easy way to restore those commits with the help of reflog.

The reflog is an ordered list of commits which HEAD has ever pointed to. This is often referred to as a safety net which means that you shouldn’t worry about that your data will ever be lost even if you change git history with git reset or a wrong rebase. Git reflog allows you to almost always recover your project’s history. I say “almost” because reflog doesn’t store entries forever, but only for configured period of time.

To see a list of HEAD positions in you repository, you can run a command:

$ git reflog

If you want to restore your project’s history after changing it, all you need to do is to find required position of HEAD in reflog output and use command:

$ git reset --hard <head_position>

Suppose we made a couple of commits creating and changing a file nginx.rb

But we decided that we didn’t need this file in our project at all so we ran command:

$ git reset --hard 673a3b2 # reset to the commit before nginx.rb was introduced

All the commits made after the commit we passed as an argement would be erased from the project’s history:

If after some time we realize that we actually need this nginx.rb file in our project and the commits we just reset were not so bad after all, we can restore our commits very easily.

First, we look into reflog:

We see that at this moment (HEAD@{0}) HEAD is pointed at 673a3b2 commit to which we made a reset. And before that (HEAD@{1}) HEAD was looking at c9236fe which was the last commit that we made before we did the reset.

So we just need to tell Git to reset us again to that commit:

$ git reset --hard c9236fe # or HEAD@{1}

Our commit history is restored and the HEAD is pointing to the commit we specified:

Master Git (part II). Viewing and undoing commits.

2017-03-27T00:00:00Z

We continue to talk about git and in this post we’ll talk about a few more things that make people confused - viewing old versions of your files and undoing commits.

How to view the old version of my repository?

To view a previous version of your repository, you can use the following command:

$ git checkout <commit> # you need to provide a commit hash or tag

Checking out an old commit is a read-only operation. So you it won’t affect the current state of your repository. Although you can create a separate branch to make your changes on the previous commit permanent.

After you’re finished viewing the old version of your repository, you can move the HEAD back to the tip of your branch and load the current state of your repository:

git checkout <current_branch_name>

How to revert back to an old version of a file?

git checkout <commit> <file>

This turns the <file> that resides in the working directory into an exact copy of the one from <commit> and adds it to the staging area. You can then re-commit the old version as you would any other file. This basically serves as a way to revert back to an old version of an individual file.

How to undo a commit?

There are tree commands that you can use.

In cases when you forgot to include files in the previous commit or wish to rewrite its message, git commit --amend is the command to use. This command lets you combine files in your index with the previous commit.

Let’s imagine that we made a commit but forgot to include a certain file.

To edit the commit and add the forgotten file to it without changing the message (--no-edit), we first stage the file and then run git commit --amend command:

Most of the times, I use git commit --amend command to simply change the message of the previous commit.

Important thing you should know before using this command is that amending removes the previous commit and replaces it with a new commit. So you should never use amend commits that have been pushed to a public repository. Otherwise, to other team members who have based their work on amended commit, it will look like the bases of their work disappeared from the project history.

Another command to undo a commit is git revert

$ git revert <commit>

git revert command undoes the changes introduced by the commit and creates a new commit with the resulting content. It is considered to be a “safe” way of undoing changes because this operation doesn’t change the commit history.

This is the command you want to use when you need to fix a specific public (shared among others) commit in your history.

Let’s say we made a commit and introduced some bad code into our application.

The content of our application file now looks like this:

So now if we want to revert the “bad” commit we run this command:

$ git revert c1196bd # hash of a bad commit

If we check git log now, we’ll see that a new commit was created:

And this commit reverted the changes of the “bad” commit:

Because git revert doesn’t change the commit history, it’s used for undoing commits which were already published to the public repository.

For reverting local changes there is the get reset command. It returns your project to the state of a commit you specify while removing all the commits made after that commit.

# undo all the commits after <commit>, but don't touch the index nor remove the changes made to the repository after the <commit>
$ git reset --soft <commit>
# undo all the commits after <commit>, reset the index, but don't remove the changes made to the repository after the <commit>
$ git reset <commit>
# Completely undo all commits and changes after the <commit>
$ git reset --hard <commit>

Git reset alters the history, which is why it’s considered to be dangerous and should never be used to reset commits that have already been published to the origin (which means other team members may base their work on that).

Remember revert is meant to undo public commits, reset is for local changes which were not pushed to a public repository.

If you’re experimenting on something locally and made a few commits that you wish to undo git reset is the command to use.

For example, let’s reset the 3 last commits to completely remove our experimenting with git revert command:

$ git reset --hard HEAD~3

If we now look for the file app.py that was added 3 commits ago (see above screenshots), we are not going to find it, because our repository was restored to the state before this file was created.

DevOps experience

Tailscale. A VPN you forget is even there.

What Tailscale is

Setting it up on a server

ACLs are actually good

What about my private network?

The honest trade-offs

GitHub Actions for AWS deployments. A small, sane setup.

The old way

The new way: OIDC

The trust policy

The workflow

The bit I keep forgetting

OpenTofu vs Terraform. Should I switch?

What actually happened

The practical answer

Should you switch?

Running both

What I actually do

IAM roles for service accounts on EKS. A small primer.

The pieces

The trust policy

The service account

A Terraform snippet

The thing people forget

Kubernetes in 2026. What changed since the last post here.

The control plane is not your problem

Helm is not the only answer

GitOps is just how this works now

Node autoscaling without Cluster Autoscaler

Ingress is now Gateway API

Observability: OTel everywhere

A few smaller things

Docker multi-stage builds and BuildKit. Shaving off the megabytes.

The naïve version

Multi-stage builds

BuildKit

A quick checklist

A few git commands I wish I knew sooner.

git bisect — when did this break?

git worktree — checking out two branches at once

git sparse-checkout — only the bits you need

That is it

From Disqus to giscus. Comments without the bloat.

The cleanup

What I considered

giscus in five minutes

The trade-offs

How to test Terraform built-in functions locally.

Terraform console

Output variables

How to build a CI/CD pipeline using Kubernetes, Gitlab CI, and Helm.

Deploy a Kubernetes cluster

Configure Kubernetes cluster

Deploy Gitlab CI and Kube-Lego

Create a new group of projects

Describe a CI/CD pipeline for each project

Launch pipeline for each service

Accessing the Raddit application

Destroy the playground

How to visualize your workflow with GitHub projects using AWS Lambda.

The problem

GitHub marketplace

AWS Lambda

Githubotik

Example

Conclusion

Master Linux CLI (part I). Useful Linux commands.

tee

pbcopy (Mac) or xclip (Linux)

watch

script & scriptreplay

jq

env

What are Docker OS images and why would I want to use them in my Dockerfile?

Iterm2 + Tmux = Awesome

AWS Roles. When and how I can use them?

Example

Conclusions

Master Git (part V). Change commits in your history. Interactive rebase.

`git bisect` — when did this break?

`git worktree` — checking out two branches at once

`git sparse-checkout` — only the bits you need