利用Qemu-4.0虚拟ARM64实验平台
https://chasinglulu.github.io/2019/07/27/%E5%88%A9%E7%94%A8Qemu-4-0%E8%99%9A%E6%8B%9FARM64%E5%AE%9E%E9%AA%8C%E5%B9%B3%E5%8F%B0/
How to launch ARM aarch64 VM with QEMU from scratch.
The below instructions will allow for bringing up an ARM VM from scratch.
It is worth mentioning that our work includes automating all of the below, resulting in a more streamlined option to building and launching the VM. See the article on QEMU aarch64 VMs for more details.
To launch an aarch64 VM we first need to install a few dependencies, including QEMU and the qemu-efi-aarch64 package, which includes the efi firmware.
apt-get install qemu-system-arm
apt-get install qemu-efi-aarch64
apt-get install qemu-utils
Create the flash images with the correct sizes.
dd if=/dev/zero of=flash1.img bs=1M count=64
dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=flash0.img conv=notrunc
Download the image you want to boot.
For our example we use an Ubuntu installer.
wget http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/main/installer-arm64/current/images/netboot/mini.iso
Create the empty Ubuntu image file we will install Ubuntu into.
We will use 20 gigabytes for this file.
qemu-img create ubuntu-image.img 20G
Start QEMU with the installer.
qemu-system-aarch64 -nographic -machine virt,gic-version=max -m 512M -cpu max -smp 4 \
-netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet \
-drive file=ubuntu-image.img,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 \
-drive file=mini.iso,if=none,id=drive1,cache=writeback -device virtio-blk,drive=drive1,bootindex=1 \
-drive file=flash0.img,format=raw,if=pflash -drive file=flash1.img,format=raw,if=pflash
Follow the instructions to install Ubuntu to the ubuntu-image.img file.
Once the install is finished you can exit QEMU with -a x.
Then restart QEMU without the installer image with the following command.
qemu-system-aarch64 -nographic -machine virt,gic-version=max -m 512M -cpu max -smp 4 \
-netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet \
-drive file=ubuntu-image.img,if=none,id=drive0,cache=writeback -device virtio-blk,drive=drive0,bootindex=0 \
-drive file=flash0.img,format=raw,if=pflash -drive file=flash1.img,format=raw,if=pflash
qemu启动参数需要添加：
-fsdev local,security_model=passthrough,id=fsdev0,path=/tmp/share -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare
然后提示：
'virtio-9p-pci' is not a valid device model name
需要重新编译qemu，编译时添加额外的configure参数：
--enable-virtfs
编译安装qemu后，再运行，可以了。
在guest中挂载host共享的目录：
mkdir /tmp/host_files
mount -t 9p -o trans=virtio,version=9p2000.L hostshare /tmp/host_files
提示：
mount: unknown filesystem type '9p'
需要在kernel中添加9p的支持：
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P_DEBUG=y (Optional)
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
然后就可以使用了，如果再加上virtio console的支持，那么既有shell，又可以传输文件，基本可以代替adb的常用功能了。
参考：
qemu启动命令：http://www.linux-kvm.org/page/9p_virtio
kernel配置：http://wiki.qemu.org/Documentation/9psetup
qemu添加configure选项：https://groups.google.com/forum/#!topic/coreos-dev/MjhL3tOOAVM
mkdir /tmp/host_files
mount -t 9p -o trans=virtio,version=9p2000.L hostshare /tmp/host_files
/usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel" -initrd /usr/share/kata-containers/binary/kata-containers.img -fsdev local,security_model=passthrough,id=fsdev0,path=./nfsroot -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare
/usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none
https://futurewei-cloud.github.io/ARM-Datacenter/qemu/how-to-launch-aarch64-vm/
root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=./mysock qemu-system-aarch64: -chardev socket,id=charconsole0,path=./mysock: Failed to connect socket ./mysock: Connection refused root@ubuntu:/home/ubuntu# ls dirA dirB dirC mysock nfsroot qemu.sh qemu.sh.bak server.py txt root@ubuntu:/home/ubuntu#
qemu-system-aarch64: root=/dev/pmem0p1: Could not open 'root=/dev/pmem0p1': No such file or directory root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none
root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/pmem0p1 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none
root@ubuntu:/home/ubuntu# ps -elf | grep 20462 7 S root 20462 20399 4 80 0 - 801176 sys_po 20:25 ? 00:00:02 /usr/share/kata-containers/binary/qemu-system-aarch64 -name sandbox-3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc -uuid 2939253a-c41f-4b7d-bf0c-3fc4ca555d85 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -qmp unix:/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=257176M -device pcie-pci-bridge,bus=pcie.0,id=pcie-bridge-0,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/binary/kata-containers.img,size=184549376 -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/shared,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -append console=hvc0 console=hvc1 iommu.passthrough=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug panic=1 nr_cpus=64 agent.use_vsock=false systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug agent.debug_console -pidfile /run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/pid -D /run/vc/vm/3e1da8e133933ccda5b51f651591da44bd52646ffa6bd7e8f8fd47fded34fdfc/qemu.log -smp 1,cores=1,threads=1,sockets=64,maxcpus=64 1 S root 20466 2 0 80 0 - 0 vhost_ 20:25 ? 00:00:00 [vhost-20462] 0 S root 20513 20131 0 80 0 - 1097 pipe_w 20:26 pts/1 00:00:00 grep --color=auto 20462 root@ubuntu:/home/ubuntu# pidof qemu root@ubuntu:/home/ubuntu# pidof qemu-system-aarch64 20462 root@ubuntu:/home/ubuntu#
root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/ram0 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none -device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3 --enable-kvm qemu-system-aarch64: -device vhost-vsock-pci,id=vhost-vsock-pci0,guest-cid=3: device is modern-only, use disable-legacy=on root@ubuntu:/home/ubuntu#
qemu-system-x86_64 \ -nographic \ -nodefaults \ -L . \ -machine virt,accel=kvm,kernel_irqchip,nofw \ -smp sockets=1,cpus=4,cores=2,maxcpus=8 \ -cpu host \ -m 2G,slots=3,maxmem=16G \ -kernel /home/sebastien/workloads/linux_4_17/linux-4.17.14/arch/x86/boot/compressed/vmlinux.bin \ -append 'console=hvc0 single iommu=false root=/dev/ram0 pci=lastbus=0' \ -device virtio-serial-pci,id=virtio-serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev stdio,id=charconsole0 \ -device sysbus-debugcon,iobase=0x402,chardev=debugcon -chardev file,path=/tmp/debug-log,id=debugcon \ -monitor telnet:127.0.0.1:55555,server,nowait \ -qmp unix:/run/virtcontainers/pods/wH9OFIZOLD5jtSs3PNht/monitor.sock,server,nowait \ -device pcie-root-port,id=rp1,addr=6,bus=pcie.0,multifunction=on
root@ubuntu:/home/ubuntu# /usr/share/kata-containers/binary/qemu-system-aarch64 -machine virt,usb=off,accel=kvm,gic-version=host,nvdimm -cpu host,pmu=off -smp 2 -m 4096M -kernel /usr/share/kata-containers/binary/vmlinuz-5.4.34-88 -nographic -append "root=/dev/pmem0p1 rw rootfstype=ext4 console=ttyAMA0 ignore_loglevel systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none agent.log=debug agent.log=debug" -initrd /usr/share/kata-containers/binary/kata-containers.img -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=./nfsroot,security_model=none -device virtio-serial-pci,id=virtio-serial0 -monitor telnet:127.0.0.1:55555,server,nowait
-monitor dev
Redirect the monitor to host device dev (same devices as the serial port). The default device is vc in graphical mode and stdio in non graphical mode. Use -monitor none to disable the default monitor.
-qmp dev
Like -monitor but opens in ’control’ mode.
-qmp-pretty dev
Like -qmp but uses pretty JSON formatting.
-mon [chardev=]name[,mode=readline|control][,pretty[=on|off]]
Setup monitor on chardev name. pretty turns on JSON pretty printing easing human reading and debugging.
-debugcon dev
Redirect the debug console to host device dev (same devices as the serial port). The debug console is an I/O port which is typically port 0xe9; writing to that I/O port sends output to this device. The default device is vc in graphical mode and stdio in non graphical mode.
# telnet 监听
qemu-system-i386 -monitor telnet:127.0.0.1:55555,server,nowait;
echo system_powerdown |telnet 127.0.0.1 55555
# tcp 监听
qemu-system-i386 -monitor tcp:127.0.0.1:55555,server,nowait;
echo 'info kvm' |nc -N 127.0.0.1 55555
echo system_powerdown |nc -N 127.0.0.1 55555
# unix
# -qmp JSON 协议
qemu-system-i386 monitor unix:qemu-monitor-socket,server,nowai
socat -,echo=0,icanon=0 unix-connect:qemu-monitor-socket
echo "info status" | socat - unix-connect:qemu-monitor-socket
echo "info status" | socat - unix-connect:qemu-monitor-socket | tail --lines=+2 | grep -v '^(qemu)'
# 查看版本
info version
# qemu -vnc :1,password
# 给 VNC 设置密码
change vnc password
# qemu -S
# 默认不启动 cpu, 手动启动
c