In continuation of my previous article, I'm diving deeper into Alpine Linux to be able to customize my stuff on the USB armory.

As I'm learning more and more about how things comes together, I've become fascinated by how they do things. I think it's ingenious how they implement some kind of immutable system by having rootfs in RAM in an initramfs-esque style. For the uninitiated: initramfs gets loaded along with the kernel by the bootloader like usual; the difference is the init script sets up the rootfs from scratch by creating the bare directory structure, then installing the packages from scratch, and finally unpacking the overlay tarball on top of it to apply the user's configuration.

Customizing Overlay

As mentioned above, the overlay is simply a tarball. On boot, after the packages are installed, this gets unpacked. The content of this tarball follows the system's directory structure, but only has the files that are user-defined. As the result, the files included get added on top of the bare system, with the preexisting files getting replaced.

I started by combining alpine.ovlapk.tar.gz and headless.ovlapk.tar.gz, since I wanted the best of both worlds.

After making sure I'd get a usable system with this, I carried on with further customizations.

Trimming Unnecessary Fat

The headless overlay comes with a script that looks for user provided configuration files in the micro SD root, such as the interface. It also does a bunch of extra stuff like setting the hostname, configuring the network interface, and starting up the SSH server (including loading the keys).

I don't need all that since I'd be baking all that I need directly onto the custom overlay of mine. The first thing I did was removing the script, as well as the init stuff for it.

Networking

With the headless script gone, I then had to make sure by myself that:

  • the USB ethernet gadget gets enabled
  • the network interface gets initiated
  • the SSH server gets started

The first one is taken care of with modules boot parameter on extlinux.conf, and a .conf file inside etc/modprobe.d.

For the network interface, I moved interface from the micro SD root to etc/network/ of the overlay. I also ensured networking service gets started by creating a symlink on etc/runlevel/default/networking of the overlay that points to /etc/init.d/networking.

For the SSH server, I added the package openssh-server into the package set by creating a line with it on etc/apk/world.

I then edit sshd_config with:

Include /etc/ssh/sshd_config.d/*.conf

AuthenticationMethods none
PermitEmptyPasswords yes
PermitRootLogin yes

AllowTcpForwarding no
GatewayPorts no
X11Forwarding no

# override default of no subsystems
Subsystem   sftp    internal-sftp

Most of it were already there. I just removed all the lines commented out and added these three lines:

AuthenticationMethods none
PermitEmptyPasswords yes
PermitRootLogin yes

I then generated SSH keys with:

ssh-keygen -f etc/ssh/ssh_host_rsa_key
ssh-keygen -t ed25519 -f etc/ssh/ssh_host_ed25519_key

Just like the networking service, I ensured sshd service gets started by linking etc/runlevel/default/sshd of the overlay to /etc/init.d/sshd.

I simply packed the overlay by turning it to a .tar.gz -- .ovlapk needs to be added to the file name right before the .tar.gz, so for this instance, I was naming it testing.ovlapk.tar.gz. I then copied the tarball into the root of my micro SD and ensure there's no other overlay.

Now, when I connect the USB Armory to my PC, a network interface appears. After ensuring the manual IP setting, I was able to SSH in.

Ethernet Gadget MAC Adress

There was a little issue, though. The MAC address of the network interface seemed to get randomized everytime the USB Armory got connected (i.e. turned on). As a result, my network setting menu became polluted, since it would create a new device entry everytime.

What I knew so far was that the MAC address can be set thru options upon loading the module with modprobe.

The module configuration that comes with the headless overlay, located in etc/modprobe.d/headless_gadget.conf, has g_cdc. I'm using g_ether, as according to USB Armory documentation.

I tried:

  1. switching back and forth between g_ether and g_cdc on modprobe
  2. switching back and forth between g_ether and g_cdc on extlinux
  3. g_cdc on modprobe, g_ether on extlinux
  4. having g_ether on etc/modules instead on extlinux.conf
  5. use_eem=0 on modprobe

In the end, it was #5 that really solved this issue, with g_ether on everything instead of g_cdc. I also kept #4 since having the ethernet device too early is neither necessary, nor desirable.

As a last touch, I customized the MAC address. I simply generated a random MAC address for each dev_addr and host_addr with macchanger command on my PC. It does need a valid interface, so I chose loopback to not affect any actual interface.

macchanger -r lo

I then copied the outputed New MAC onto appropriate parts of the modprobe file, which I took the liberty to rename as usbarmory.conf.

In the end, I got:

options g_ether use_eem=0 dev_addr=d2:2f:ee:24:f1:20 host_addr=72:aa:f2:c3:f7:e7

Creating Local APK Repo

After satisfied with the basic system config, I wanted to see if I could meddle with the installed package even more.

I started with curl, nano, and wget. Just like openssh-server in the previous section, I added them to etc/apk/world.

When I booted my system, I was greeted with errors telling these packages were not available on the serial terminal. That's when I knew I needed to expand my local repo to include the packages in the world list along with their dependencies.

Packages & Index

I was able to login normally. So I first started with setup-apkcache and setup-apkrepos so I can get packages from the main repo.

After I installed the packages I wanted, I looked into copying from /var/cache/apks to /media/mmcblk0p1/apks. However, I realized these packages has strange hash-like hex numbers appended to them. I didn't want that, so instead I fetched the packages raw and generated the index from it.

apk fetch --no-cache --recursive $(cat /etc/apk/world)
apk index *.apk -o APKINDEX.tar.gz

I replaced the stock local repo with the one I just put together.

Again, I naively tried to boot off with this, and again I was greeted by error. This time, it wasn't letting me in at all.

Turned out, APKINDEX.tar.gz needs to be signed. After some reading, I learned abuild-sign command is used to achieve exactly this. The command in question is included in abuild package.

I put back the stock local repo so I could boot normally. From there, I redid the setup-apkcache and setup-apkrepos so I could get the package from the online repo with apk add abuild.

I then generated the key with the command abuild-keygen, with which I got a key pair. The private key was then used to sign the index.

abuild-sign -k <generated> /media/mmcblk0p1/apks/armv7.new/APKINDEX.tar.gz

Finally, I copied the public key to etc/apk/keys, and kept the private key in a safe place.

An Issue

Upon boot, I encountered error on the apk setup part. On serial console, the login prompt appeared, but trying to log in with my usual credential didn't work.

I went on the UART and found these error messages.

ERROR: alpine-baselayout-data-3.4.3-r2: package mentioned in index not found (try 'apk update')
ERROR: alpine-baselayout-3.4.3-r2: package mentioned in index not found (try 'apk update')
ERROR: alpine-release-3.19.1-r0: package mentioned in index not found (try 'apk update')
ERROR: ca-certificates-bundle-20240226-r0: package mentioned in index not found (try 'apk update')
ERROR: mdev-conf-4.6-r0: package mentioned in index not found (try 'apk update')
ERROR: busybox-mdev-openrc-1.36.1-r15: package mentioned in index not found (try 'apk update')
ERROR: busybox-openrc-1.36.1-r15: package mentioned in index not found (try 'apk update')
ERROR: libc-utils-0.7.2-r5: package mentioned in index not found (try 'apk update')
ERROR: alpine-base-3.19.1-r0: package mentioned in index not found (try 'apk update')
ERROR: ncurses-terminfo-base-6.4_p20231125-r0: package mentioned in index not found (try 'apk update')
ERROR: openssh-server-common-9.6_p1-r0: package mentioned in index not found (try 'apk update')
ERROR: openssh-server-common-openrc-9.6_p1-r0: package mentioned in index not found (try 'apk update')

Out of the many packages to be installed, why only these? I checked the packages in question to exist.

I was then greeted by a login prompt. I tried logging in as root, but I couldn't get thru.

Reproducing The Issue

I wasn't able to do anything on it since it didn't even let me in. The first thing that popped up in my mind was to have a working system and try to reproduce the issue from there. That was I'd be able to take a look at the inside of things. After taking a look at the init script, I got some idea of what to do.

I first put my packages in a tarball, called apks.new.tar, to ensure it wouldn't get detected as a local repo at boot. I then put back the stock local repo. With the working system I put back together, I executed:

tar -xf /media/mmcblk0p1/apks.new.tar -C /tmp
mkdir /tmp/a/
apk add --root /tmp/a/ --initdb --quiet
apk add --root /tmp/a/ -X /tmp/apks/ --keys-dir /etc/apk/keys --initramfs-diskless-boot --clean-protected --overlay-from-stdin $(cat /etc/apk/world) < /media/mmcblk0p1/acrost-armory.apkovl.tar.gz

And the same issue appeared!

Solution

After reading thru the entries on APKINDEX, I noticed some entries has no arch, and that these entries are more or less the very packages that failed.

Turned out the arch needs to be the same as all the others. Otherwise, the packages won't be recognized somehow.

To achieve this, I used --rewrite-arch armv7 option on the apk index command.

Hence the command:

apk index -v --no-cache --rewrite-arch armv7 -d 3.19.1 -o APKINDEX.tar.gz *.apk

After putting my repo back, with the new APKINDEX.tar.gz, the system boots successfully!

Generating initramfs

Last time I tried the mkinitfs tool, the system failed to boot. After learning more about Alpine Linux, I became more confident in trying again.

First, I installed mkinitfs with:

apk add mkinitfs

Then I added g_ether and ledtrig-heartbeat modules that are not included. I simply created a 'feature' called usbarmory, which includes these two modules. You can actually add other stuff like libraries or scripts into a feature, but these two modules would do.

cat > /etc/mkinitfs/features.d/usbarmory.modules << EOF
kernel/drivers/usb/gadget/legacy/g_ether.ko
kernel/drivers/leds/trigger/ledtrig-heartbeat.ko
EOF

I removed my generated key from /etc/apk/keys/

rm /etc/apk/keys/<generated_key>  # because we have the key on the overlay

To run the mkinitfs command:

mkinitfs -F "base ext4 mmc usb usbarmory" -o /media/mmcblk0p1/boot/initramfs-testing

Then I appended on extlinux/extlinux.conf:

LABEL testing
MENU LABEL Linux testing
KERNEL /boot/zImage-6.6.24-0-usbarmory
INITRD /boot/initramfs-testing
FDTDIR /boot/dtbs-usbarmory
APPEND modules=loop,squashfs,sd-mod,usb-storage,ledtrig_heartbeat quiet modloop=/boot/modloop-usbarmory

Without forgetting to change DEFAULT to testing on extlinux/extlinux.conf, I booted the system.

To my delight, it booted just fine!

After testing everything to work, I moved initramfs-testing into initramfs-6.6.24-0-usbarmory, and removed the testing entry from extlinux.conf to finalize things.

Ref:

  • https://wiki.alpinelinux.org/wiki/Manually_editing_a_existing_apkovl
  • https://github.com/usbarmory/usbarmory/wiki/Host-communication