Tag: Libvirt

linux – security implications of libvirt socket access

In most Linux distributions it is possible to manage virtual machines as a regular user if that user has access to /run/libvirt/libvirt-sock and is a member of the libvirt group.

I understand that we cannot give a regular user access to root powers without any security implications. I’m not entirely clear though what these security implications are, or in other words, I don’t know how much trust I need to place in users that are libvirt members.

The system in question is a single user desktop, and the concern is full privilege escalation to root. An attacker running with user privileges should not be able to abuse libvirt for modifying the host system in arbitrary ways.
It is understood that such an attacker could create and otherwise manage virtual machines at free will.

kvm virtualization – Using the libvirt terraform provider to attach to system wide network bridge

I’m trying to use the libvirt kvm terraform provider (https://github.com/dmacvicar/terraform-provider-libvirt) to connect an example virtual machine to an existing network bridge I’ve created. I’m using the simple example from here; https://github.com/fabianlee/terraform-libvirt-ubuntu-examples.git . Here’s the tf, the bridge details are below that; any suggestions would be gratefully received.

# variables that can be overriden
variable "hostname" { default = "simple" }
variable "domain" { default = "example.com" }
variable "memoryMB" { default = 1024*1 }
variable "cpu" { default = 1 }

# instance the provider
provider "libvirt" {
  uri = "qemu:///system"
}

# fetch the latest ubuntu release image from their mirrors
resource "libvirt_volume" "os_image" {
  name = "${var.hostname}-os_image"
  pool = "default"
  source = "https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img"
  format = "qcow2"
}

# Use CloudInit ISO to add ssh-key to the instance
resource "libvirt_cloudinit_disk" "commoninit" {
          name = "${var.hostname}-commoninit.iso"
          pool = "default"
          user_data = data.template_file.user_data.rendered
          network_config = data.template_file.network_config.rendered
}

resource "libvirt_network" "vm_network" {
  name = "test_net"
  mode = "bridge"
  bridge = "br0"
}


data "template_file" "user_data" {
  template = file("${path.module}/cloud_init.cfg")
  vars = {
    hostname = var.hostname
    fqdn = "${var.hostname}.${var.domain}"
  }
}

data "template_file" "network_config" {
  template = file("${path.module}/network_config_dhcp.cfg")
}


# Create the machine
resource "libvirt_domain" "domain-ubuntu" {
  name = var.hostname
  memory = var.memoryMB
  vcpu = var.cpu

  disk {
       volume_id = libvirt_volume.os_image.id
  }
  network_interface {
       network_name = "default"
  }

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  # IMPORTANT
  # Ubuntu can hang is a isa-serial is not present at boot time.
  # If you find your CPU 100% and never is available this is why
  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = "true"
  }
}

terraform { 
  required_version = ">= 0.12"
}

output "ips" {
  # show IP, run 'terraform refresh' if not populated
  value = libvirt_domain.domain-ubuntu.*.network_interface.0.addresses
}

Bridge br0:

5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1d:7d:0d:2a:9d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.250/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fde6:4511:f54:0:21d:7dff:fe0d:2a9d/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::21d:7dff:fe0d:2a9d/64 scope link 
       valid_lft forever preferred_lft forever

iptables – libvirt with qemu/kvm guest, guest can ssh to host and vice versa, but failed to samba or ftp to host

I am running libvirt/qemu-kvm on Fedora32, guest OS is CentOS7.

I use ‘nat’ mode virtual networking.

root@fedora ~)# virsh net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>36ca4070-160a-47bf-b35e-aa7bee028ec1</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:e1:1e:c3'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

On host I can ssh to guest by its ip (192.168.122.230).

On guest, I can access internet, also can ssh to my host,
but failed to access samba and ftp on my host.

For example, I type ‘smbclient -L ‘192.168.122.1’‘ on guest,
host ‘tcpdump -i vnet0‘ shows:

10:03:00.267931 IP 192.168.122.230.57754 > 192.168.122.1.microsoft-ds: Flags (S), seq 1417555984, win 29200, options (mss 1460,sackOK,TS val 4294755489 ecr 0,nop,wscale 7), length 0
10:03:00.267977 IP 192.168.122.1 > 192.168.122.230: ICMP 192.168.122.1 tcp port microsoft-ds unreachable, length 68
10:03:00.273271 IP 192.168.122.230.39152 > 192.168.122.1.netbios-ssn: Flags (S), seq 2454440184, win 29200, options (mss 1460,sackOK,TS val 4294755494 ecr 0,nop,wscale 7), length 0
10:03:00.273290 IP 192.168.122.1 > 192.168.122.230: ICMP 192.168.122.1 tcp port netbios-ssn unreachable, length 68

And ‘smbclient’ eventually reports ‘* do_connect: Connection to 192.168.122.1 failed (Error NT_STATUS_CONNECTION_REFUSED)*’.

In case of ‘ftp’, it is similar to ‘samba’.

0:06:11.030486 IP 192.168.122.230.44748 > 192.168.122.1.ftp: Flags (S), seq 4205484033, win 29200, options (mss 1460,sackOK,TS val 4294946254 ecr 0,nop,wscale 7), length 0
10:06:11.030539 IP 192.168.122.1 > 192.168.122.230: ICMP 192.168.122.1 tcp port ftp unreachable, length 68

I am sure on guest, firewall is turned off, and I can samba to host from other machine in lan.

I checked host ‘iptables -L -nv ‘ and ‘iptables -L -nv -t nat’, no packet got ‘REJECT’ed or ‘DROP’ed.

They look like this:

# iptables -L -nv 
Chain INPUT (policy ACCEPT 56760 packets, 31M bytes)
 pkts bytes target     prot opt in     out     source               destination         
68394   45M LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
19326   23M LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
19326   23M LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 9344 1092K LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 19706 packets, 2824K bytes)
 pkts bytes target     prot opt in     out     source               destination         
28243 3880K LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_FWI (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 9982   22M ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 9344 1092K ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_INP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  102  6959 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    9  3028 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    9  3004 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:68

and

# iptables -L -nv -t nat
Chain PREROUTING (policy ACCEPT 6314 packets, 5976K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 4463 packets, 5827K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 546 packets, 73524 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 526 packets, 69524 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1910  218K LIBVIRT_PRT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain LIBVIRT_PRT (1 references)
 pkts bytes target     prot opt in     out     source               destination
   13  1359 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24
    0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255
   87  4628 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
  192 19180 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24

Am I missing something? What could be the cause?
Thanks.

networking – libvirt on fedora, qemu/kvm guest, guest can ssh to host, but failed to samba or ftp to host

I am running libvirt/qemu-kvm on Fedora32, guest OS is win10 with spice-guest-tool in use.

I use ‘nat’ mode virtual networking.

root@fedora ~)# virsh net-dumpxml default
<network connections='1'>
  <name>default</name>
  <uuid>36ca4070-160a-47bf-b35e-aa7bee028ec1</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:e1:1e:c3'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

While guest is running, ‘brctl show‘ seems fine.

(root@fedora ~)# brctl show
bridge name bridge id       STP enabled interfaces
virbr0      8000.525400e11ec3   yes     virbr0-nic
                                        vnet0

On host I can ping guest by its ip (192.168.122.159).

On guest, I can access internet, also can ssh to my host,
but failed to access samba and ftp on my host.

For example, I type ‘net view 192.168.122.1‘ on guest,
host ‘tcpdump -i vnet0‘ shows:

15:47:39.041395 IP 192.168.122.159.49717 > fedora.bear.microsoft-ds: Flags (S), seq 160880283, win 64240, options (mss 1460,nop,wscale 8,nop,nop,sackOK), length 0
15:47:39.041526 IP fedora.bear > 192.168.122.159: ICMP fedora.bear tcp port microsoft-ds unreachable, length 60

And ‘net view’ eventually reports ‘System error 53: network path not found‘.
I also checked ‘iptables -L -v‘ (too verbose to paste here), no one got ‘REJECT’ed.

In case of ‘ftp’, it is similar to ‘samba’.

15:54:13.232366 IP 192.168.122.159.49721 > fedora.bear.ftp: Flags (S), seq 669575524, win 8192, options (mss 1460,nop,wscale 0,nop,nop,sackOK), length 0
15:54:13.232468 IP fedora.bear > 192.168.122.159: ICMP fedora.bear tcp port ftp unreachable, length 60

It seems that host can not send package back to guest.

Am I missing something? What could be the cause?
Thanks.

kvm virtualization – How do remove the default storage pool from a libvirt hypervisor, so that even after libvirtd restarts there is NO storage pool

I want to remove the default storage pool from my virt-manager AND NOT HAVE IT COME BACK BY ITSELF, EVER. I can destroy it and undefine it all I want, but when i restart libvirtd (for me thats “sudo systemctl restart libvirtd” in an arch linux terminal window), and restart virt-manager, the default storage pool is back, just like Frankenstein.

I don’t want a storage pool of any kind. I simply want to move from the dual-boot I have now (arch linux and windows) to running the two OS simultaneously. I intend to provision two physical disk partitions on the host to be disks on the guest, and I can do this via the xml that defines the domain.

Or am i required to have a storage pool no matter what?

hardlink – libvirt qemu AppArmor 9p hard links

I am using libvirt with qemu on a debian host. A virtual machine has a defined 9p mount point:

The default apparmor setting (which is apparently created by virt-aa-helper) does not allow me to create hard links on 9p volume.

I made it work by adding the following line to /etc/apparmor.d/abstractions/libvirt-qemu

  "/mnt/pool/share/**" rwl,

This works but you have the following problems:

  1. It allows everybody hosts to read / write to this directory, not just the host I need
  2. Requires editing a configuration file that is updated regularly, which is probably not a good idea because it makes it difficult to update the Debian package
  3. It is not configured in the libvirt xml file, which makes portability difficult and an additional step

Is there a better way?

Libvirt Migration Host Name: Server Error

I have configured two libvirt nodes with the following host names:

  • mycompany-hv-01.example.tld
  • mycompany-hv-02.example.tld

The names are declared in a public DNS and can be resolved (public IP).

When I try to migrate a guest from one host to another:

root@mycompany-hv-02:~# virsh migrate prout qemu+ssh://mycompany-hv-01.example.tld/system --offline --persistent
error: internal error: hostname on destination resolved to localhost, but migration requires an FQDN

The error is the same when I try a live migration.

I know this is not exactly the same error, but I tried the tips on this page. My DNS already works, so I tried to force the resolution by adding entries to my /etc/hosts on both hosts, but that doesn't work.

The following solution works:

virsh migrate prout qemu+ssh://mycompany-hv-02.example.tld/system tcp://mycompany-hv-02.example.tld --offline --persistent

I tried to define manually migration_host in /etc/libvirt/qemu.conf but I received the error:

configuration file syntax error: migration_host must not be the address of the local machine: mycompany-hv-01.example.tld

Do I miss something?

Kvm virtualization: how can I make my libvirt / KVM guest see all IPv4 / UDP multicast traffic?

I have a problem with IPv4 / UDP multicast traffic that is not completely visible from a KVM guest.

The guest has a dedicated NIC that is attached via MacVTap. Both the host operating system and the guest operating system are Ubuntu 18.04. This is the network configuration of the VM:

(Note that I obfuscated the actual MAC address).

Since I have enabled trustGuestRxFilters, mDNS is working fine. However, there is still some multicast traffic that I cannot see.

This is the command that generates problematic UDP multicast traffic:

raspivid -ae 40,0xff -a 1036 -t 0 -w 1280 -h 720 -ih -fps 30 -mm spot 
  -o udp://239.255.0.1:5000

This creates a constant video transmission of approximately 400 KB / s. Here I am deliberately using multicast, so several hosts on the network can play or record the transmission (without requiring that the source computer, which is connected via WiFi, send multiple transmissions). It is assumed that the host of the KVM guest analyzes the transmission and records it every time there is movement in the video.

Here is the problem: All hosts that are directly Connected to the network (= not a KVM guest) can receive UDP traffic, even the KVM host itself. However, the KVM guest cannot, only sees very few packages:

sudo timeout 20 tcpdump -i ens3 host 239.255.0.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
20:53:10.361355 IP vogelhaus.internal.example.com.48146 > 239.255.0.1.5000: UDP, length 4096
#
# omitted 5 lines
#
20:53:12.081881 IP vogelhaus.internal.example.com.48146 > 239.255.0.1.5000: UDP, length 4096

7 packets captured
16 packets received by filter
9 packets dropped by kernel

These are definitely not enough packages for a 400 KB / s video stream that runs for 20 seconds. When I do the same on another host that is directly connected, I get the expected results:

sudo timeout 20 tcpdump -i enp1s0f0 host 239.255.0.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0f0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:54:41.264709 IP vogelhaus.internal.example.com > 239.255.0.1: udp
#
# ... many, many more! ...
#
20:55:00.912257 IP vogelhaus.internal.example.com > 239.255.0.1: udp
20:55:00.912446 IP vogelhaus.internal.example.com.48146 > 239.255.0.1.5000: UDP, bad length 4096 > 1472

7205 packets captured
7231 packets received by filter
26 packets dropped by kernel

The operating system on the KVM host is Ubuntu 18.04. QEMU version:

QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.21)

Any idea what prevents the KVM guest from seeing all the traffic?

Kvm virtualization: libvirt and macvtap problem in Arch Linux

I am using Arch Linux (updated) with QEMU-KVM, libvirt and virt-manager as front. I have several virtual machines, but so far only one is running at a time. The virtual machine I try to put to work is in Debian 10, but I also have a Kali and a CentOS 7 with the same problem when I try similar things.

The interface I am trying to use for macvtap is a wireless card (on a Thinkpad T580 laptop) connected to a Wi-Fi access point (WPA2).

I am trying to configure a macvtap interface to connect the wlp4s0 connection on my host to one of my virtual machines. To do that, I am using virt-manager. I tried bridge and VEPA mode on the macvtap, and tried all types of interfaces (virtual hardware) on the VM, without success, since there is no network connection. However, NAT mode works well on all virtual machines.

Libvirt puts the device (wlp4s0 on the host) in promiscuous mode, although ip-link does not show it (the mark in / sys / devices / … is changing, and dmesg says something about entering promiscuous mode).

When starting Wireshark and pinging the gateway (with a fixed IP) from the VM, I see the ARP request on the host in macvtap and in wlp4s0, but there is no response.

When using dhcp, dhclient gets no response.

I can provide more information is necessary. If you have any idea what is causing that, I will gladly listen to your suggestions!

Linux – QEMU / KVM / libvirt macvtap VEPA does not work – ARP request not forwarded

Hi, I have been struggling to make the guest network work when I use a macvtap in VEPA mode between two virtual machines on a host. I've spent hours (days) searching on Google without joy. Does this network configuration really work?

I created the vtap using KVM Manager by adding a NIC, selecting the "macvtap" network source, the VEPA source mode, the device model: virtio.

The configuration looks like this (mac address):

vm3-62                                  vm2-62
----------                           ------------
eth1: 172.15.62.105            eth1:  172.15.62.205
(52:54:00:08:9d:8b)            (52::54:00:8a:b1:0f)
           +                               +
           |                               |
                                          /
                           host          /
            macvtap1                  macvtap0
          (52:54:00:08:9d:8b)      (52:54:00:8a:b1:0f)
                                          /
                                         /
                                        /
                           bond1.62
                     (98:03:9b:2d:91:a2) 
                              |
                           bond1
                             |
                       NIC port 1 and 2 (active/passive config)

Not sure if the above will format well, if the previous diagram was not formatted here is a JPEG

The host NIC is connected to a cisco nexus 9000, which I configured for the 802.1Qbg reflective relay.

In vm2-62 when I try to ping 172.15.62.105, I get the unreachable destination Host.

When I use tcpdump on the host, I can see the ARP request of vm2-62 looking for the mac for 172.15.62.105 (vm3-62). I can see the request in macvtap0, and in bond1.62 and in bond1, but NOT in macvtap1.

If I manually add the ARP inputs in vm3-62 and vm2-62, the ping works fine, so I think the reflective relay on the switch is set correctly.

It seems that the switch is not retrieving the ARP request or that I need to do something in Linux to enable bond1.62 to forward the ARP request to macvtap1.

Any ideas? .

Thank you