Overview
After massaging the blog’s theme to have a few more elements I wanted it was time for the next step - automatic docker builds and publishing to my Gitea’s package system. I recently learned that Gitea supports several packaging systems, such as Chef, Helm, and Docker. For a complete list, check out this page. The idea sounded great: develop my blog, commit changes, and have a docker image created and stored on my server so I can later deploy the new version via CD tools.
Gitea Runners on Kubernetes
I already have a gitea runner available on my k8s homelab that I use for automatic updates of my pi-hole cluster. It seemed logical to extend this service to also be able to package docker images. However, what I thought would be a very straightforward workflow turned into a major headache due to my runner being a “Docker in Docker” deployment. The primary usecase for gitea runners appears to be as standalone servers, your workstation, or a virtual machine. Unfortunately, running a runner within a Docker environment like Kubernetes or flat Docker has a few extra quirks that took a some scouring to resolve.
Docker in Docker
The Gitea act-runner system closely follows Github’s actions system - both systems utilize a runner that executes steps/actions in docker images. Because of how the gitea act-runner image behaves, the runner and its Docker container are run in separate containers in the same pod instead of one all-in-one runner that has docker installed locally. At the surface level there is no issue with this setup, it creates one major issue - the docker actions assume that docker is available locally and /var/run/docker exists and can be interacted with.
The workflow failed giving a straightforward error:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
The error became clear: the docker actions were trying to hit the local docker socket when instead they should be using the docker websocket exposed by the runner’s docker container at localhost:2376. The issue seemed simple enough to fix: simply pass the same environment variables that the runner uses to communicate with the docker container:
- name: Set up Docker BuildX
uses: docker/setup-buildx-action@v3
with:
config-inline: |
[registry."repo.zerosla.com"]
env:
- name: DOCKER_HOST
value: tcp://localhost:2376
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: "1"
For whatever reason, the values were not being applied correctly and the workflow actions were still trying to target the socket. Unfortunately every how-to I could find appeared to use non-DinD runners that had locally available docker sockets and had no issue. Finally I struck gold: Gitea forum user mariusrugan provided a working gitea-runner-dind-kubernetes deployment
The solution appeared to be a few simple steps:
- Modify my worker’s deployment yaml to include a config.yaml file for the runner, allowing me to further customize the runner.
- Include default options for actions the runner spawns that properly sets the docker parameters.
- Ensure that the shared certificate directory was being passed to action containers as well.
The Fix
My final deployment yaml ended up looking like this:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: act-runner-vol
namespace: gitea
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: longhorn
---
apiVersion: v1
data:
# The registration token can be obtained from the web UI, API or command-line.
# You can also set a pre-defined global runner registration token for the Gitea instance via
# `GITEA_RUNNER_REGISTRATION_TOKEN`/`GITEA_RUNNER_REGISTRATION_TOKEN_FILE` environment variable.
token: "base64 encoded token"
kind: Secret
metadata:
name: runner-secret
namespace: gitea
type: Opaque
---
apiVersion: v1
kind: ConfigMap
metadata:
name: runner-config
namespace: gitea
annotations:
reloader.stakater.com/auto: "true"
data:
config.yaml: |-
log:
# The level of logging, can be trace, debug, info, warn, error, fatal
level: debug
runner:
# Where to store the registration result.
file: .runner
# Execute how many tasks concurrently at the same time.
capacity: 1
# Extra environment variables to run jobs.
envs:
A_TEST_ENV_NAME_1: a_test_env_value_1
A_TEST_ENV_NAME_2: a_test_env_value_2
# Extra environment variables to run jobs from a file.
# It will be ignored if it's empty or the file doesn't exist.
env_file: .env
# The timeout for a job to be finished.
# Please note that the Gitea instance also has a timeout (3h by default) for the job.
# So the job could be stopped by the Gitea instance if it's timeout is shorter than this.
timeout: 30m
# Whether skip verifying the TLS certificate of the Gitea instance.
insecure: false
# The timeout for fetching the job from the Gitea instance.
fetch_timeout: 5s
# The interval for fetching the job from the Gitea instance.
fetch_interval: 2s
# The labels of a runner are used to determine which jobs the runner can run, and how to run them.
# Like: "macos-arm64:host" or "ubuntu-latest:docker://gitea/runner-images:ubuntu-latest"
# Find more images provided by Gitea at https://gitea.com/gitea/runner-images .
# If it's empty when registering, it will ask for inputting labels.
# If it's empty when execute `daemon`, will use labels in `.runner` file.
labels:
- "ubuntu-latest:docker://gitea/runner-images:ubuntu-latest"
- "ubuntu-22.04:docker://gitea/runner-images:ubuntu-22.04"
- "ubuntu-20.04:docker://gitea/runner-images:ubuntu-20.04"
cache:
# Enable cache server to use actions/cache.
enabled: true
# The directory to store the cache data.
# If it's empty, the cache data will be stored in $HOME/.cache/actcache.
dir: ""
# The host of the cache server.
# It's not for the address to listen, but the address to connect from job containers.
# So 0.0.0.0 is a bad choice, leave it empty to detect automatically.
host: ""
# The port of the cache server.
# 0 means to use a random available port.
port: 0
# The external cache server URL. Valid only when enable is true.
# If it's specified, act_runner will use this URL as the ACTIONS_CACHE_URL rather than start a server by itself.
# The URL should generally end with "/".
external_server: ""
container:
# Specifies the network to which the container will connect.
# Could be host, bridge or the name of a custom network.
# If it's empty, act_runner will create a network automatically.
network: ""
# Whether to use privileged mode or not when launching task containers (privileged mode is required for Docker-in-Docker).
privileged: false
# And other options to be used when the container is started (eg, --add-host=my.gitea.url:host-gateway).
options: |
--add-host=docker:host-gateway
-v "/certs:/certs"
-e "DOCKER_HOST=tcp://docker:2376/"
-e "DOCKER_TLS_CERTDIR=/certs"
-e "DOCKER_TLS_VERIFY=1"
-e "DOCKER_CERT_PATH=/certs/client"
# The parent directory of a job's working directory.
# NOTE: There is no need to add the first '/' of the path as act_runner will add it automatically.
# If the path starts with '/', the '/' will be trimmed.
# For example, if the parent directory is /path/to/my/dir, workdir_parent should be path/to/my/dir
# If it's empty, /workspace will be used.
# workdir_parent:
# Volumes (including bind mounts) can be mounted to containers. Glob syntax is supported, see https://github.com/gobwas/glob
# You can specify multiple volumes. If the sequence is empty, no volumes can be mounted.
# For example, if you only allow containers to mount the `data` volume and all the json files in `/src`, you should change the config to:
# valid_volumes:
# - data
# - /src/*.json
# If you want to allow any volume, please use the following configuration:
# valid_volumes:
# - '**'
valid_volumes:
- /certs
# overrides the docker client host with the specified one.
# If it's empty, act_runner will find an available docker host automatically.
# If it's "-", act_runner will find an available docker host automatically, but the docker host won't be mounted to the job containers and service containers.
# If it's not empty or "-", the specified docker host will be used. An error will be returned if it doesn't work.
# docker_host: ""
# Pull docker image(s) even if already present
# force_pull: true
# Rebuild docker image(s) even if already present
# force_rebuild: false
host:
# The parent directory of a job's working directory.
# If it's empty, $HOME/.cache/act/ will be used.
# workdir_parent:
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: act-runner
name: act-runner
namespace: gitea
spec:
replicas: 1
selector:
matchLabels:
app: act-runner
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: act-runner
spec:
restartPolicy: Always
volumes:
- name: docker-certs
emptyDir: {}
- name: config
configMap:
name: runner-config
- name: runner-data
persistentVolumeClaim:
claimName: act-runner-vol
containers:
- name: runner
image: gitea/act_runner:nightly
command: ["sh", "-c", "while ! nc -z localhost 2376 </dev/null; do echo 'waiting for docker daemon...'; sleep 5; done; /sbin/tini -- run.sh"]
env:
- name: DOCKER_HOST
value: tcp://localhost:2376
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: "1"
- name: CONFIG_FILE
value: "/config.yaml"
- name: GITEA_INSTANCE_URL
value: https://repo.zerosla.com
- name: GITEA_RUNNER_REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
name: runner-secret
key: token
- name: GITEA_RUNNER_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
volumeMounts:
- name: docker-certs
mountPath: /certs
- name: runner-data
mountPath: /data
- name: config
mountPath: /config.yaml
subPath: config.yaml
- name: daemon
image: docker:28-dind
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
securityContext:
privileged: true
volumeMounts:
- name: docker-certs
mountPath: /certs
The only real divergences from mariusrugan’s gist is that I provided a list of default options, thereby ensuring that all docker actions can properly talk to the docker daemon’s websocket.
Submodules
Finally, my containers were building and being published, but I hit another snag between the workers and my workstation. Hugo was failing to build itself because it was missing layout files. Silly me forgot that I deployed my branch of the aafu theme as a submodule, not part of the blog’s repo. This was an easy fix - simply instruct the checkout action to recurse submodules:
diff --git a/.gitea/workflows/docker-publish.yml b/.gitea/workflows/docker-publish.yml
index 880d770..484f05e 100644
--- a/.gitea/workflows/docker-publish.yml
+++ b/.gitea/workflows/docker-publish.yml
@@ -8,13 +8,8 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v3
+ with:
+ submodules: recursive
# - name: Set up QEMU
# uses: docker/setup-qemu-action@v2
Putting it all together
My final workflow looks something like this:
on:
push:
branches:
- '*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
submodules: recursive
# - name: Set up QEMU
# uses: docker/setup-qemu-action@v2
- name: Set up Docker BuildX
uses: docker/setup-buildx-action@v3
with:
config-inline: |
[registry."repo.zerosla.com"]
- name: Log into Gitea Registry
uses: docker/login-action@v2
with:
registry: repo.zerosla.com
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v4
with:
context: .
file: ./Dockerfile
platforms: |
linux/amd64
push: true
tags: |
repo.zerosla.com/${{ secrets.REGISTRY_USER }}/hugoblog:${{ github.ref_name }}
repo.zerosla.com/${{ secrets.REGISTRY_USER }}/hugoblog:${{ github.sha }}
With my build pipeline fully functioning, I can finally move onto the next phase: continuous deployment into my homelab. I hope to cover this in Part 3 of building this blog!