Commit Graph

1035 Commits

Author SHA1 Message Date
Jonathon Fernyhough
f502b56ec2 Create zsync control file for delta downloads 2021-03-14 17:42:54 +00:00
nl6720
bc67933af1
Support EROFS
EROFS, like Squashfs, is a read-only file system. It can be used to store airootfs in an image file.
Its advantage is the support for POSIX ACLs. EROFS downside is that currently it only supports LZ4 compression (LZMA support is not yet fully implemented).

A difference from Squashfs is that, EROFS stores change time (ctime) not modification time (mtime). The reverse is true for Squashfs.

Implements https://gitlab.archlinux.org/archlinux/archiso/-/issues/59
2021-03-09 16:25:45 +02:00
nl6720
711ab4cd1e
archiso/initcpio/hooks/archiso: remove redundant /sfs/ from airootfs mount point
Remove /run/archiso/bootmnt directory if nothing is mounted there. An empty directory is just confusing.
2021-03-09 16:25:45 +02:00
David Runge
652ad4deed releng: Add usbmuxd to list of packages
configs/releng/packages.x86_64:
Add usbmuxd to list of packages, so that users have the option to use iOS devices out-of-the-box for data connection
during installation.

Fixes #99
2021-02-16 21:43:46 +00:00
Kristian Klausen
03ac49f64e Remove haveged
haveged was added 8 years ago[1] to increase entropy and presumably to
prevent entropy starvation.

A few things has changed since, most notable:
* the kernel actively tries to add entropy (jitter entropy)[2][3][4][5]
* /dev/random no longer blocks after CRNG initialization[6][7]

[1] d7e790d ("Initialize pacman keyring on bootup")
[2] 3f2dc2798b
[3] 50ee7529ec
[4] https://lore.kernel.org/lkml/alpine.DEB.2.21.1909290010500.2636@nanos.tec.linutronix.de/T/
[5] https://lwn.net/Articles/800509/
[6] 30c08efec8
[7] https://lwn.net/Articles/808575/

Fix #98
2021-02-16 16:33:06 +00:00
nl6720
97f510df81
.gitlab-ci.yml: use correct units for image size description
`du --block-size=MiB` (and `du -m`) returns mebibytes not megabytes.

Additionally, shorten the du command. `du --block-size=MiB` is the same as `du -m`.
2021-02-06 15:35:55 +02:00
David Runge
f0ef2f3caf
Add changelog file
CHANGELOG.rst:
Add file to track changes (at least for v51).
2021-02-01 09:59:49 +01:00
David Runge
b588c52665
build-host.sh: Style and syntax fixes
.gitlab/ci/build-host.sh:
Set shebang to /usr/bin/env bash to be more portable/flexible.
Turn all posix statements ([]) to bash style statements ([[]]), as we are using bash.
Terminate the list of parameters to rm or cp with --.

Replace the implementation of finding a local ISO to use with one that relies on a sorted list of potential images.

Use virtio-net-pci for networking with qemu.
Set the cow_spacesize to 4G for the archiso environment.

Use --needed in the call to pacman to not re-install already up-to-date targets.
Attempt a full system upgrade (but ignore the kernel).
Increase the timeout for when installing packages to the archiso environment using pacman to 120s, as a system upgrade
is being done as well.

Use systemctl poweroff -i to shut down the virtual machine as it is more future proof and robust.
2021-01-31 23:46:51 +01:00
David Runge
428bf47370
Add build stage for continuous integration
.gitlab-ci.yml:
Add a build stage to the gitlab CI, that facilitates the scripts below .gitlab/ci/, building the baseline and releng
profiles in parallel.
Distinguish the use-case in which builds are done for master, schedules and tags in a secure environment and any other
where builds just have to be fast (for ensuring nothing is broken).

Use MiB as block size for the du call when generating data for the metrics file.
2021-01-31 23:25:22 +01:00
David Runge
2a24429404
Add continuous integration scripts to linting
Makefile:
Add scripts below .gitlab/ci/ to the lint target of the Makefile.
2021-01-31 16:27:41 +01:00
David Runge
07239499a6
Add scripts for continuous integration
.gitlab/ci/build-host.sh:
Add script to be run in a container with access to qemu.
It is a slight modification of arch-boxes' build-host.sh script to cater to the specific archiso requirements.

.gitlab/ci/build-inside-vm.sh:
Add script to be run in virtualized environment, established by build-host.sh.
This script builds the actual archiso profiles and creates checksum for the resulting image files.
2021-01-31 16:23:04 +01:00
nl6720
f9a7b206a5
configs/releng: move locale-gen from customize_airootfs.sh to a pacman hook
This finally removes customize_airootfs.sh from releng.

Fixes https://gitlab.archlinux.org/archlinux/archiso/-/issues/21 .
2021-01-31 11:34:56 +02:00
nl6720
a46c74087f
config/releng: remove pacman hooks specific to ISO build process from airootfs after they run
This works around https://bugs.archlinux.org/task/49347 .
Leaving the hooks in the airootfs image will result in it being run when pacstrap is run in the live environment. This should not happen as they are intended for the ISO build process only.

Fixes https://gitlab.archlinux.org/archlinux/archiso/-/issues/91 .
2021-01-31 09:55:25 +02:00
Sven-Hendrik Haase
767c09569d
Update my email in AUTHORS 2021-01-31 02:12:36 +01:00
Michael Vorburger
dd51457d09 rm un-used set_image() dupe of check_image() in run_archiso.sh 2021-01-31 01:02:53 +00:00
Michael Vorburger.ch
23c5d52bd8 add vorburger to AUTHORS.rst 2021-01-31 01:00:18 +00:00
nl6720
0f20a11bb7
Support setting more variables in profiledef.sh and rework the way overrides are applied
- Apply overrides before validating the options.
- Parse all paths with realpath. Fixes https://gitlab.archlinux.org/archlinux/archiso/-/issues/84 .
2021-01-29 00:01:24 +02:00
Michael Vorburger
e7306a309c add cloud-init to baseline 2021-01-27 14:16:43 +00:00
Michael Vorburger
744b8f13ed make baseline have working ethernet networking, like releng 2021-01-27 14:16:43 +00:00
Michael Vorburger
94dd194c22 add SSH server to baseline
inspired by https://wiki.archlinux.org/index.php/Archiso#Prepare_an_ISO_for_an_installation_via_SSH
2021-01-27 14:16:43 +00:00
Michael Vorburger
221a9211e5 use same airootfs_image_tool_options in baseline as in releng 2021-01-27 14:16:43 +00:00
nl6720
4f4047a3f8
configs/releng: move the mirror uncommenting sed command from customize_airootfs.sh to a pacman hook
After pacman-mirrorlist is installed, /etc/pacman.d/hooks/uncomment-mirrors.hook will run a sed command which uncomments all Server lines in /etc/pacman.d/mirrorlist.
This brings us another step closer to the complete removal of customize_airootfs.sh.

Related to https://gitlab.archlinux.org/archlinux/archiso/-/issues/21 .
2021-01-27 00:07:45 +02:00
Christian Hesse
a2c8dd3173 archiso_pxe_common: remove resolv.conf before copy
Booting via PXE we want to keep our DNS configuration. So remove
/etc/resolv.conf in new root before copying the current file.

Without this systemd-resolved fallback nameservers are used and we see an
error message when the root ships a symbolic link to systemd-resolved's
stub-resolv.conf:

cp: not writing through dangling symlink '/new_root/etc/resolv.conf'
2021-01-26 21:59:01 +00:00
Christian Hesse
b30d1cad9b mkarchiso: add version information
To date the iso version was used for iso volume information and iso file name.
In my custom builds I do use it a lot more:

* Inside the root fs: The system knows about its own version. I use this to:
  -> report the version to a server (poor man's inventory)
  -> let the system update itself

* On the iso fs: The files are served via rsync, running systems transfer
  version file first to check for available update.

* A grub environment file on the iso fs: Booting the iso from grub allows
  to create cow directory per version:

    loopback loop archlinux.iso
    load_env -f (loop)/arch/grubenv
    linux (loop)/arch/boot/x86_64/vmlinuz-linux ... \
        cow_directory=archlinux/${VERSION} ...

So let's just create these files.
2021-01-26 10:00:38 +01:00
Christian Hesse
d153b48613 mkarchiso: fix typos 2021-01-23 21:04:17 +01:00
Michael Vorburger
0ae5cd961e lint ALL (future) scripts/*.sh, not just run_archiso.sh 2021-01-23 19:23:38 +00:00
Michael Vorburger
608d1100c8 add -c => -cdrom cloud-init.iso support to run_archiso.sh
Co-authored-by: nl6720 <nl6720@gmail.com>
2021-01-23 16:43:55 +00:00
Michael Vorburger
88304719d0 add cloud-init files to .gitignore (and loosen out/ and work/) 2021-01-23 16:07:36 +00:00
Michael Vorburger
955343925b make run_archiso forward VM's port 22 to host 60022, for easy SSH testing 2021-01-23 16:02:50 +00:00
Michael Vorburger
833cbd5e72 enable SSH server in releng profile 2021-01-23 15:54:31 +00:00
Michael Vorburger
9b03e0b08a add cloud-init to releng 2021-01-23 15:54:31 +00:00
Michael Vorburger
e556491c27
add VNC support to run_archiso.sh
Usage: Launch run_archiso.sh -v ..., and then use a VNC viewer
(e.g. from https://wiki.archlinux.org/index.php/List_of_applications/Internet#Remote_desktop)
to connect (typically to `localhost`) on the default VNC port (5900).

This enables using run_archiso in a "headless" session; e.g. when SSH logged in
to the CLI of a VM, without a local display attached.  This is handy e.g. when
playing https://en.wikipedia.org/wiki/Inception and running an archlinux*.iso
on any non-Arch (say Fedora workstation), on which one built a new ISO, that you
then "run_archiso", inside which you could build another ISO, which you could
itself start inside the nested VM... ;-)

Jokes apart, this could also be used to run automated CI/CD tests of the built ISO,
which is particularly interesting in combination with the cloud-init support;
see https://wiki.archlinux.org/index.php/Cloud-init.

see https://bugs.archlinux.org/task/69142
2021-01-23 14:50:06 +01:00
nl6720
64091a1802
Combine sed commands to reduce file writes 2021-01-07 14:33:28 +02:00
nl6720
a178ff44bc
profiledef.sh: Remove www. from archlinux.org
See https://lists.archlinux.org/pipermail/arch-devops/2020-December/000474.html .
2020-12-26 20:11:06 +02:00
nl6720
8ba2870829
configs/releng/syslinux/archiso_pxe-linux.cfg: add missing /boot to initrd file path
Fixes https://bugs.archlinux.org/task/68803 .
2020-12-01 07:12:31 +02:00
David Runge
c10004dfec
Fix issues with file ownerships/modes
archiso/mkarchiso:
Make sure to always compare absolute paths in `_make_custom_airootfs()` (as `realpath` is used).
Remove `echo` calls that prevent the setting of actual file ownerships and modes.

configs/releng/profiledef.sh:
Set file mode of /root/.automated_script.sh to 755.

Fixes #82
2020-11-30 21:48:08 +01:00
nl6720
863247d0a6
Keep all SYSLINUX files in /syslinux
This gets rid of the duplicate ldlinux.c32 and the useless isolinux.cfg which only points to syslinux.cfg.

Implements https://gitlab.archlinux.org/archlinux/archiso/-/issues/46 .
2020-11-30 16:00:15 +02:00
nl6720
183ae52792
Prevent path traversal outside of $airootfs_dir 2020-11-30 09:21:35 +02:00
nl6720
42d9e4f983
Allow specifying ownership and mode of custom airootfs files and directories
profiledef.sh can now contain an associative array called file_permissions which can be used to set custom ownership and mode of custom airootfs files. The array's keys contain the path and the value is a colon separated list of owner UID, owner GID and access mode.
For example:

    file_permissions=(
      ["/etc/shadow"]="0:0:400"
    )

This means that mkarchiso now copies airootfs files (and directores) without permissions and anything that should be owned by a user other than root and/or if the mode should be something other than 644 for files and 755 for directories must to be listed in ${file_permission[@]} in profiledef.sh.

Fixes https://gitlab.archlinux.org/archlinux/archiso/-/issues/61 .
2020-11-30 08:46:24 +02:00
fdupoux
2c99df5c9b Reset network interfaces at the end of the PXE boot to allow DHCP to run 2020-11-18 21:16:27 +00:00
David Runge
3160db0e9e
Fix evaluation bugs in mkarchiso
archiso/mkarchiso:
Guard the call to `_mksignature()` in `_prepare_airootfs_image()` by an if statement.
Using the `&&` logic leads to `_prepare_airootfs_image()` evaluating to false if `$gpg_key` is not set.

Add `_msg_info()` calls to `_set_override()` which prevent the function from evaluating to false if no override is
being done. Additionally this is great for debugging purposes.

Add `_msg_info()` calls to `_read_profile()` (which is great for debugging purposes).

Fixes #81
2020-11-18 19:24:02 +01:00
Sven-Hendrik Haase
6c397136fd Use official archlinux Docker image
archlinux/base is being deprecated anyway.
2020-11-17 11:35:46 +00:00
nl6720
96ac5e2454
mkarchiso: add xorrisofs options from boot mode specific functions instead of hardcoding them in _build_iso 2020-11-14 15:36:45 +02:00
nl6720
4dfb473748
mkarchiso: validate profile right after reading it
Fixes https://gitlab.archlinux.org/archlinux/archiso/-/issues/76 .
2020-11-14 15:29:51 +02:00
nl6720
57d510fe7f
mkarchiso: general cleanup and simplification
- Remove remnants of the now removed legacy commands.
- Improve readability by getting rid of some "if" statements when performing string comparisons.
- Rename functions to make their purpose more clear.
- Move some conditions from functions to their invocations.
2020-11-14 14:49:33 +02:00
Sven-Hendrik Haase
bb91fd52d9
Use better xz options
Default: 732M
With these options: 675M
2020-11-03 21:17:20 +01:00
nl6720
cc169d7e31
configs/releng/packages.x86_64: add fatresize, gpart and tmux
- fatresize is a utility to resize FAT filesystems using libparted.
- gpart is partition table rescue/guessing tool.
- tmux is a terminal multiplexer. Requested in https://bugs.archlinux.org/task/68252 .
2020-11-01 19:46:32 +02:00
David Runge
40ae3ba97c
Set syslinux bootloader timeout to 15s
configs/releng/syslinux/archiso_sys.cfg:
Set the syslinux bootloader timeout to 15s.
In !97 it has accidentally been set to 1.5s.
2020-10-31 19:12:44 +01:00
David Runge
ee6c5faa86
Set bootloader timeouts to 15s
configs/releng/efiboot/loader/loader.conf,
configs/releng/syslinux/archiso_sys.cfg:

Set the bootloader timeout to 15s, as they have been set to an overly generous 30s in !79.

Fixes #80
2020-10-31 17:53:40 +01:00
fdupoux
9f16862acd Configure the image type and image creation options using profiles (#54) 2020-10-30 21:33:08 +00:00