Podman rootless and virtualbox shared folders using subuid
by Dr Giuseppe Insana
Dear Podman users and developers
I wonder if you have encountered a similar problem to mine and whether
you may have a solution.
Description of the problem:
I cannot run containers where non-root users of the container (e.g. uid
101 of nginx for a nginx container) have possibility to write to a
mounted folder from virtualbox. Even when mounting the shared folder
with full writing permissions (dmode/fmode 0777), the new files created
appear as belonging to nobody (uid 65534). This prevents any application
which creates new files or directories (e.g. wordpress).
The setup:
* I have a linux virtual machine (Alpine 3.11.6, tested on both 5.4.27
and 5.4.34 kernels) running over a linux host (Ubuntu 18.04.4 LTS) on
which I create and run rootless pods and containers.
* virtualbox version: 5.2.34
* podman version: tried both 1.8.2 and 1.9.1
* shared folder and its contents are owned by 1001:1001 on the host
* the folder is mounted in the vm with the command /sbin/mount.vboxsf -o
rw,uid=100100,gid=1001,dmode=0775,fmode=0664 wpcontent
/media/sf_wpcontent
* the user running the container in the vm has id/gid 1001:1001 and
subuid 100000:65536
Investigation:
I'll paste the result of commands run at three levels: the host, the vm
and the container.
* This is on the underlying host:
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
$ touch uploads/sites/3/2020/host-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
-rw-rw-r-- 1 1001 1001 0 Apr 22 15:10 host-touched
$ sudo chown 100100:100100 uploads/sites/3/2020/host-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 2 1001 1001 20480 Jan 31 16:05 01
drwxrwxr-x 2 1001 1001 12288 Feb 18 08:34 02
drwxrwxr-x 2 1001 1001 4096 Apr 21 16:48 04
-rw-rw-r-- 1 100100 100100 0 Apr 22 15:10 host-touched
* This is in the VM:
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
$ touch /media/sf_wpcontent/uploads/sites/3/2020/vm-touched
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:28 vm-touched
$ chown 100100:100100
/media/sf_wpcontent/uploads/sites/3/2020/vm-touched
$ ls -ln /media/sf_wpcontent/uploads/sites/3/2020
total 36
drwxrwxr-x 1 100100 1001 20480 Jan 31 16:05 01
drwxrwxr-x 1 100100 1001 12288 Feb 18 08:34 02
drwxrwxr-x 1 100100 1001 4096 Apr 21 16:48 04
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 100100 1001 0 Apr 22 15:28 vm-touched
This is in the container:
# ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
# touch uploads/sites/3/2020/cont-touched
$ ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:31 cont-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
# chown 100100:100100 uploads/sites/3/2020/cont-
touched
chown: uploads/sites/3/2020/cont-touched: Invalid argument
# chown nginx:nginx uploads/sites/3/2020/cont-to
uched
# ls -ln uploads/sites/3/2020
total 36
drwxrwxr-x 1 101 0 20480 Jan 31 16:05 01
drwxrwxr-x 1 101 0 12288 Feb 18 08:34 02
drwxrwxr-x 1 101 0 4096 Apr 21 16:48 04
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:31 cont-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:10 host-touched
-r-xr-xr-x 1 65534 1001 0 Apr 22 15:28 vm-touched
rm -f uploads/sites/3/2020/cont-touched
rm: can't remove 'uploads/sites/3/2020/cont-touched': Value too large
for data type
The behaviour is inconsistent, for example:
in the vm:
$ ls -ln uploads/sites/
total 16
drwxrwxr-x 1 100100 1001 4096 Apr 22 15:52 2
drwxrwxr-x 1 100100 1001 4096 Apr 22 08:52 3
drwxrwxr-x 1 100100 1001 4096 Jan 21 14:30 4
drwxrwxr-x 1 100100 1001 4096 Jun 14 2017 5
inside the container:
# ls -ln uploads/sites/
total 16
drwxrwxr-x 1 65534 1001 4096 Apr 22 15:52 2
drwxrwxr-x 1 101 0 4096 Apr 22 08:52 3
drwxrwxr-x 1 65534 1001 4096 Jan 21 14:30 4
drwxrwxr-x 1 65534 1001 4096 Jun 14 2017 5
To clear the inconsistencies I can use the command (as vm's root):
# echo 3 > /proc/sys/vm/drop_caches
This appears to "correct" the problem by realigning the id of the
directories. But everytime a new file or directory is created, they are
still created as user nobody (65534).
Even for applications where no new files/directories are created (e.g.
appending to existing file), the drop_caches command would need to be
applied everytime the machine is rebooted, thus rendering the whole
system not really stable/usable.
The only two "solutions" that "work" are:
1) to have containers where the services run as root. Thus not using
subuid and files are created as the user running podman. Mounting the
shared folder with uid equal to the podman user is ok.
Obviously this is not always possible/desirable, as containers often
have a different user running the command.
2) mounting the shared folder with full permissions and without using a
subuid, i.e. like this:
/sbin/mount.vboxsf -o rw,uid=1001,gid=101,dmode=0777,fmode=0555
wpcontent /media/sf_wpcontent
this is obviously far from desirable, as it allows any user of the vm
permission to write into the folder or delete files
If you have used rootless containers writing on shared virtual folders
and have found a working solution, please let me know.
Thank you very much!
Best regards
Giuseppe
PS: I've also tried adding a user with uid/gid 100100:100100 in the vm.
This does not make any difference to the problems reported above
PPS: I've also tried changing id/gid of host folder or of the user in
the vm running podman. No changes.
4 years
Q re errors pulling a docker image
by Philip Rhoades
People,
When I run this:
# podman run \
--publish 3085:3085 \
-d \
--name coda \
codaprotocol/coda-demo:sandbox-32a.1
I get this:
ERRO[0003] Error deleting network: missing prevResult from earlier
plugin
ERRO[0003] Error while removing pod from CNI network "podman": missing
prevResult from earlier plugin
ERRO[0003] Error refreshing container
19e58fe9a7c81de33fb8833610cd9a0be3edadf867acf7acf9b3c78a32483835:
missing prevResult from earlier plugin
Trying to pull docker.io/codaprotocol/coda-demo:sandbox-32a.1...
.
.
The container starts up OK and I am about to see if it is actually doing
what it is supposed to (I am just getting started with Coda Protocol)
but should I be concerned about those errors?
Thanks,
Phil.
--
Philip Rhoades
PO Box 896
Cowra NSW 2794
Australia
E-mail: phil(a)pricom.com.au
4 years
Running systemd inside a container : systemd-logind error with degraded state
by Philipp Richter
Hello,
I was wondering if it's normal that systemd inside a container is in a
degraded state with systemd-logind not being able to start.
$ journalctl -b -a
Apr 28 14:05:13 824304d0fa93 systemd[27]: systemd-logind.service:
ProtectHostname=yes is configured, but UTS namespace setup is
prohibited (container manager?), ignoring namespace>
Apr 28 14:05:13 824304d0fa93 systemd[1]: Starting Permit User Sessions...
Apr 28 14:05:13 824304d0fa93 systemd[27]: systemd-logind.service:
Failed to apply hostname restrictions: Permission denied
Apr 28 14:05:13 824304d0fa93 systemd[27]: systemd-logind.service:
Failed at step SECCOMP spawning /usr/lib/systemd/systemd-logind:
Permission denied
Apr 28 14:05:13 824304d0fa93 systemd[1]: systemd-logind.service: Main
process exited, code=exited, status=228/SECCOMP
Apr 28 14:05:13 824304d0fa93 systemd[1]: systemd-logind.service:
Failed with result 'exit-code'.
Apr 28 14:05:13 824304d0fa93 systemd[1]: Failed to start Login Service.
Apr 28 14:05:13 824304d0fa93 systemd[1]: systemd-logind.service:
Scheduled restart job, restart counter is at 1.
Apr 28 14:05:13 824304d0fa93 systemd[1]: Stopped Login Service.
Apr 28 14:05:13 824304d0fa93 systemd[1]: Condition check resulted in
Load Kernel Module drm being skipped.
Apr 28 14:05:13 824304d0fa93 systemd[1]: Starting Login Service...
Apr 28 14:05:13 824304d0fa93 systemd[31]: systemd-logind.service:
ProtectHostname=yes is configured, but UTS namespace setup is
prohibited (container manager?), ignoring namespace>
Apr 28 14:05:13 824304d0fa93 systemd[31]: systemd-logind.service:
Failed to apply hostname restrictions: Permission denied
Apr 28 14:05:13 824304d0fa93 systemd[31]: systemd-logind.service:
Failed at step SECCOMP spawning /usr/lib/systemd/systemd-logind:
Permission denied
Apr 28 14:05:13 824304d0fa93 systemd[1]: systemd-logind.service: Main
process exited, code=exited, status=228/SECCOMP
Apr 28 14:05:13 824304d0fa93 systemd[1]: systemd-logind.service:
Failed with result 'exit-code'.
Apr 28 14:05:13 824304d0fa93 systemd[1]: Failed to start Login Service.
Then $ systemctl status gives :
● 824304d0fa93
State: degraded
Jobs: 0 queued
Failed: 1 units
Since: Tue 2020-04-28 14:05:11 UTC; 23min ago
CGroup: /
├─crun-exec
│ ├─43 bash
│ ├─70 systemctl status
│ └─71 less
├─init.scope
│ └─1 /sbin/init
└─system.slice
├─systemd-journald.service
│ └─15 /usr/lib/systemd/systemd-journald
└─dbus.service
└─26 /usr/bin/dbus-daemon --system --address=systemd:
--nofork --nopidfile --systemd-activation --syslog-only
I am starting a custom archlinux base image like this :
$ sudo podman run -d --cap-add=NET_ADMIN
localhost/popsulfr/archlinux-base /sbin/init
The output of podman info :
host:
arch: arm64
buildahVersion: 1.14.8
cgroupVersion: v2
conmon:
package: Unknown
path: /usr/bin/conmon
version: 'conmon version 2.0.15, commit:
1bddbf7051a973f4a4fecf06faa0c48e82f1e9e1'
cpus: 4
distribution:
distribution: archarm
version: unknown
eventLogger: journald
hostname: rock64
idMappings:
gidmap: null
uidmap: null
kernel: 5.6.7-1-ARCH
memFree: 3530731520
memTotal: 4097687552
ociRuntime:
name: crun
package: Unknown
path: /usr/bin/crun
version: |-
crun version 0.13
commit: e79e4de4ac16da0ce48777afb72c6241de870525
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
rootless: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 0
swapTotal: 0
uptime: 29m 39.39s
registries:
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 1
runRoot: /var/run/containers/storage
volumePath: /var/lib/containers/storage/volumes
Best Regards,
Philipp Richter
4 years
SubUID limitations?
by Gordon Messmer
Is there a limit on the size of UID ranges usable for subuid?
I'm trying to run a rootless container, and in that container use sssd
user info from Active Directory. sssd produces very large UID and GID
values, which I think is the problem that's being reported.
First of all, the host system can resolve information for a user named
build:
[gitlab-runner@dvpackerbld03 ~]$ id build
uid=1985001175(build) gid=1985001156(development)
groups=1985001156(development),987(docker)
And I can run the container that I've created, and resolve information
for that user there, too:
[gitlab-runner@dvpackerbld03 ~]$ podman run -it \
> --volume=/var/lib/sss/pipes/:/var/lib/sss/pipes/:rw \
> --volume=/etc/sssd/:/etc/sssd/:ro \
> --volume=/etc/krb5.conf:/etc/krb5.conf:ro \
> --volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro \
> --volume=/etc/nsswitch.conf:/etc/nsswitch.conf:ro \
> e1f5919f1888 id build
uid=1985001175(build) gid=1985001156(development)
groups=1985001156(development),996(docker)
But if I try to run "su build" within that container, I get an error:
[gitlab-runner@dvpackerbld03 ~]$ podman run -it \
--volume=/var/lib/sss/pipes/:/var/lib/sss/pipes/:rw \
--volume=/etc/sssd/:/etc/sssd/:ro \
--volume=/etc/krb5.conf:/etc/krb5.conf:ro \
--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro \
--volume=/etc/nsswitch.conf:/etc/nsswitch.conf:ro \
e1f5919f1888 su build -c ls
su: cannot set groups: Invalid argument
subuid and subgid are set up as:
[gitlab-runner@dvpackerbld03 ~]$ cat /etc/subuid
gitlab-runner:110000:3000000000
[gitlab-runner@dvpackerbld03 ~]$ cat /etc/subgid
gitlab-runner:110000:3000000000
4 years
Podman v2 development update
by Brent Baude
In the last few days, the Podman development team has been working to
release Podman-1.9.0. This is likely to be the last Podman-1.X release
before we transition to Podman-2.x. We have been working since
November 2019 to make a significant overhaul of Podman’s architecture.
And if we did our job correctly, most casual Podman users will not
notice a difference. We will continue to investigate and fix issues in
Podman-1.x versions but severity of the bug and priority will dictate
our response.
What some users who follow upstream development may notice is that
while we make the final push to a 2.x release, our GitHub repository
will look drastically different. For some period of time, certain
Podman commands, if built based on upstream, may not function exactly
as expected nor even exist. We already know we will need to disable
some of our CI testing framework as part of this final push until we
have a more complete Podman v2.x. We will not release Podman 2.0 until
we are satisfied that it is ready. While upstream development will be
impacted by the announced migration to Podman v2.x, you can still open
issues and contribute pull requests to the project.
As has been the standard with our project, we will remain transparent
in our development activities and try to keep our community appraised
of our progress. We are excited for some of the technical
advancements that Podman v2.x will give our users. Subsequent blog
posts will be written on those advancements and why they matter to our
users.
Brent
4 years, 1 month
Possible to customize outgoing IP in rootless container
by mh+podman＠scrit.ch
Hi
assuming a host has 2 ips: 192.168.1.1 and 192.168.1.2 (as an ip alias).
By default outgoing traffic is on 192.168.1.1 - is it possible to tell a
rootless container (actually slirp4netns), that 192.168.1.2 should be
used as the src ip?
I would assume that this would need to be done be slirp4netns, but I
wasn't able to find such an option.
thanks and best
mh
4 years, 1 month
ZFS Support
by Ingvar.Bogdahn＠itzbund.de
Hello,
I would like to know to which extent ZFS is supported as filesystem for podmain containers.
In particular, I'd like to know if it is possible to use ZFS clones as a backend for the container filesystem layering and if zfs datasets are usable for rootless containers (using zfs delegation). Is ZFS integration done transparently, i.e. are zfs dataset clones created when a new layer is created in a transparent way without explicit zfs commands?
Thank you
Best regards,
Ingvar Bogdahn
4 years, 1 month
