Forgejo Rootless Install with Podman and Ubuntu 24.04
I have recently switched from Docker to Podman , mostly because Podman’s integration with SystemD feels better to me than Docker Compose, especially with podman-quadlet . Setting up rootless Forgejo with Podman took some time figuring out so I decided to document it here.
Server preparation
Install a fresh copy of Ubuntu 24.04, update, and install Podman. The particular server I’m running is hosted at Hetzner . The server is configured with a Hetzner firewall that makes it only accessible over Nebula , you can use Tailscale if you want.
I like to use bind mounts backed by ZFS. This combination has served me well and has survived multiple server crashes. The easiest way to do this with Hetzner is to use a volume. I have run into a situation with Hetzner where a server would be unreachable but I could recover by creating a new server and moving the volume to the newly created server.
1apt update
2apt full-upgrade
3apt autoremove
4apt install podman zfsutils-linux
5
6# Reboot to start fresh, not necessary but I like to do it that way
7reboot
8
9# Set up ZFS (Disk ID will be different in your case)
10zpool create tank /dev/disk/by-id/scsi-SHC_Volume_103797893
11
12# Set some good zpool and dataset properties
13zpool set ashift=12 tank
14zfs set atime=off tank
15zfs set compression=lz4 tank
16zfs set aclmode=passthrough tank
17zfs set logbias=throughput tank
18
19# Make sure Podman is working
20podman run --rm hello-world
Create the user that all containers will run as. I’ll use “git” here because that’s the account users will connect with — I’ll explain this further later.
1useradd -m git
2
3# This is needed for the user "git" to have persistent services running without needing someone to log in
4loginctl enable-linger git
To run rootless Podman containers, users inside the container are mapped to actual users on the host, for this to work we have to use SUBUIDs and SUBGIDs, let’s look at the ones allowed for “git”.
1cat /etc/subuid /etc/subgid
2
3# Output
4git:100000:65536
5git:100000:65536
In this particular instance, “root” inside containers (UID 1) will map to UID 100000 on the host, in an Ubuntu container, “www-data” (UID 33), will be mapped to UID 100032.
Sometimes those numbers are different, an easy way to remap those SUBUIDs and SUBGIDs is as follows
1# Remove all allocated SUBUIDs and SUBGIDs and assign 200001-265535
2usermod --del-subuids 1-4294967295 --del-subgids 1-4294967295 --add-subuids 200001-265535 --add-subgids 200001-265535 git
I like to start the SUBUIDs with a “1” which makes it slightly easier to read. Root inside the container maps to UID 200001 and the www-data user maps to 200033. It has no technical difference and is just a personal preference.
Log in as “git”
For Podman to operate correctly, you must SSH to the host as the container-running user. Forgejo, however, expects the git user for Git access rather than shell logins, so direct SSH fails; luckily there’s a fix.
1# Install the machinectl command
2apt install systemd-container
3
4# Switch to the "git" user
5machinectl shell [email protected] /usr/bin/bash
Normally when I manage Podman containers in my homelab, I switch from root with the following command machinectl shell --uid=podman-user, but this will not work with Forgejo because we need to update the default shell to allow for git commands to work with Forgejo. More on that later.
MariaDB
Forgejo needs a database, you can use SQLite or PostgreSQL. MySQL/MariaDB is what I understand, so I’m sticking with that.
1# Create the required datasets (Run these commands as root, or a user that has permissions to modify the pool)
2zfs create tank/mysql
3zfs create tank/mysql/data
4zfs create tank/mysql/log
5zfs create tank/mysql/conf
6
7# Set optimizations for MySQL
8zfs set recordsize=16k tank/mysql/data
9zfs set primarycache=metadata tank/mysql/data
10zfs set recordsize=128k tank/mysql/log
11zfs set primarycache=metadata tank/mysql/log
12
13# Set the proper permissions
14# MariaDB runs as the user "999"
15chown 200999:200999 /tank/mysql/*
Why create “tank/mysql/…” instead of just “tank/mysql_data” and “tank/mysql_log”? Putting everything under “tank/mysql” allows us to recursively snapshot “tank/mysql” and get a consistent state in our snapshot. Also, if you end up running multiple MySQL/MariaDB instances, you can use “tank/mysql_one”, “tank/mysql_other” and you get better organization.
As for the required quadlet file
1# Switch to the "git" user
2machinectl shell [email protected] /usr/bin/bash
3
4# The default network doesn't have DNS, for containers to communicate using names we need to create a new one
5podman network create git
6
7mkdir -p ~/.config/containers/systemd/
8cd ~/.config/containers/systemd/
Create a file and call it “db.container”, don’t worry about “MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1”, we’ll fix that shortly.
1[Container]
2ContainerName=db
3Environment=MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1
4Environment=TZ=Asia/Amman
5Image=docker.io/mariadb:10.6
6Network=git
7# We don't want MariaDB to be publically reachable
8PublishPort=127.0.0.1:3306:3306
9Volume=/tank/mysql/data:/var/lib/mysql
10Volume=/tank/mysql/log:/var/lib/mysql_log
11Volume=/tank/mysql/conf:/etc/mysql/conf.d
12
13[Install]
14WantedBy=multi-user.target default.target
Create the MairaDB configuration file
1# /tank/mysql/conf/70-zfs.cnf
2[mysqld]
3datadir = /var/lib/mysql
4innodb_flush_log_at_trx_commit = 1 # TPCC reqs.
5innodb_log_file_size = 1G
6innodb_log_group_home_dir = /var/lib/mysql_log
7innodb_flush_neighbors = 0
8innodb_fast_shutdown = 2
9
10innodb_flush_method = fsync
11innodb_doublewrite = 0 # ZFS is transactional
12innodb_read_io_threads = 10
13innodb_write_io_threads = 10
14
15innodb_use_native_aio=0
16innodb_log_write_ahead_size=16384
17
18innodb_file_per_table=on
19performance-schema = ON
20performance_schema=ON
21
22query_cache_type = 1
23query_cache_limit = 256M
24query_cache_size = 1024M
25
26skip-external-locking
27skip-name-resolve
28
29table_definition_cache=2048
30join_buffer_size=16M
31key_buffer_size=32M
32innodb_buffer_pool_size=2G
33innodb_log_file_size=512M
34innodb_log_buffer_size=32M
Start and secure the database.
1systemctl --user daemon-reload
2systemctl --user start db.service
If you don’t get an error, check the running container
1podman container ls
With the database now running, let’s apply some security
1podman exec -it db bash
2
3# Run this inside the container
4mysql_secure_installation
As you’ve seen, we set up socket authentication without a password. This makes the database only accessible from inside the container. Next, allow management from the host.
1MariaDB [(none)]> grant all on *.* to root@'%' identified by 'some_super_secure_password' with grant option;
Configure access from the host
1apt install mariadb-client
Place the following in “/root/.my.cnf” and set the permissions to 0400
1[client]
2user=root
3host=127.0.0.1
4password=some_super_secure_password
1chmod 0400 /root/.my.cnf
2
3# Test connectivity
4mysql
Create the Forgejo database
1create database forgejo;
2grant all on forgejo.* to forgejo@'%' identified by 'some_other_super_secret_password';
Set up a reverse proxy with HTTPS
We’ll use Caddy running on the host to handle HTTPS. Since we’re using rootless Podman the containers cannot bind to ports 80 and 443 directly. There are ways to do so, but I haven’t researched the security implications.
1# Download Caddy from https://caddyserver.com/download, the one that comes with Ubuntu doesn't have Cloudflare support
2mv caddy /usr/local/bin/
3chmod 755 /usr/local/bin/caddy
4mkdir /etc/caddy
Create the following Caddyfile
1repo.yourdomain.com {
2 tls {
3 dns cloudflare {env.CLOUDFLARE_API_TOKEN}
4 }
5
6 reverse_proxy localhost:3000
7}
Run Caddy manually to test, check the output to make sure your certificate is being generated
1cd /etc/caddy
2CLOUDFLARE_API_TOKEN=MY_SUPER_SECRET_TOKEN caddy run
Create the SystemD unit file “/etc/systemd/system/caddy.service”
1[Unit]
2Description=Caddy
3Documentation=https://caddyserver.com/docs/
4After=network.target network-online.target
5Requires=network-online.target
6
7[Service]
8Type=notify
9User=www-data
10Group=www-data
11EnvironmentFile=/etc/caddy/caddy.env
12ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
13ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
14TimeoutStopSec=5s
15LimitNOFILE=1048576
16LimitNPROC=512
17PrivateTmp=true
18ProtectSystem=full
19AmbientCapabilities=CAP_NET_BIND_SERVICE
20
21# Restart configuration
22Restart=on-failure
23RestartSec=5s
24
25[Install]
26WantedBy=multi-user.target
Create the required environment file “/etc/caddy/caddy.env”
1CLOUDFLARE_API_TOKEN=MY_SUPER_SECRET_TOKEN
Run Caddy
1# Make sure /var/www is created and owned by www-data
2mkdir /var/www
3chown -R www-data:www-data /var/www
4
5systemctl daemon-reload
6systemctl enable --now caddy
Forgejo
With the server ready, it’s time to configure Forgejo. First we need to set up storage.
1zfs create tank/forgejo
2
3# The service runs under UID 1000 in the container
4chown 201000:201000 /tank/forgejo
Create the following quadlet file “~/.config/containers/systemd/forgejo.container”
1[Unit]
2Description=Forgejo
3# Only start Forgejo once the database is up
4After=db.service
5Wants=db.service
6
7[Container]
8ContainerName=forgejo
9Image=codeberg.org/forgejo/forgejo:13-rootless
10PublishPort=127.0.0.1:3000:3000
11Network=git
12Volume=/tank/forgejo:/var/lib/gitea
13Volume=/etc/timezone:/etc/timezone:ro
14Volume=/etc/localtime:/etc/localtime:ro
15
16[Install]
17WantedBy=multi-user.target default.target
Here you will see an example of how Podman is better than Docker, with Docker compose, service dependencies is only honored when running docker compose up, but unlike Podman/SystemD, startup order is not guaranteed when you reboot the host. If you use Nginx and need to reference another container by name, this is much simpler.
Start and secure the database.
1systemctl --user daemon-reload
2systemctl --user start forgejo.service
Now access your instance using https://repo.yourdomain.com
Configure Forgejo to your liking, keep “SSH server port” set to 2222, and click “Install Forgejo”.
Setting up SSH access for git
Right now our repo is accessible over HTTPS, but for SSH, the URL isn’t pretty (and doesn’t work, for now)
We need to setup SSH passthrough for the user “git”, edit “/etc/ssh/sshd_config” and add the below to the end of the file:
1Match User git
2 AuthorizedKeysCommandUser git
3 AuthorizedKeysCommand /usr/bin/podman exec -i forgejo /usr/local/bin/gitea keys -e git -u %u -t %t -k %k
Reload sshd
1systemctl reload ssh
This allows Forgejo to handle SSH key authentication, when a user logs in, sshd asks Forgejo for the list of allowed public keys and if the user’s matches one of them, they are let in.
Edit git’s shell; create “/usr/local/bin/git-shell”
1#!/usr/bin/env bash
2
3/usr/bin/podman exec -i --env SSH_ORIGINAL_COMMAND="$SSH_ORIGINAL_COMMAND" forgejo sh "$@"
Make it executable and set it as the shell for “git”
1chmod 755 /usr/local/bin/git-shell
2chsh -s /usr/local/bin/git-shell git
Update Forgejo to know it’s running under Podman instead of Docker, edit “/tank/forgejo/custom/conf/app.ini” and update the following keys
1[server]
2; Change this from 2222 to 22
3SSH_PORT = 22
4
5; Add this
6SSH_AUTHORIZED_KEYS_COMMAND_TEMPLATE = {{.AppPath}} --config={{.CustomConf}} serv key-{{.Key.ID}}
Restart Forgejo
1machinectl shell [email protected] /usr/bin/bash
2
3systemctl --user restart forgejo
And now you can use Git operations with SSH normally.
About Me
Dev gone Ops gone DevOps. Any views expressed on this blog are mine alone and do not necessarily reflect the views of my employer.
Recent Posts