1 目的
在centos7下使用qemu搭建arm模擬器。
參考博客：
centos下利用qemu搭建arm模擬器 - 寒水司天 - 博客園 (cnblogs.com)
從零使用qemu模擬器搭建arm運行環境_海楓的專欄-CSDN博客_qemu模擬arm
用Qemu模擬vexpress-a9 （一） --- 搭建Linux kernel調試環境 - 摩斯電碼 - 博客園 (cnblogs.com)
2 環境
centos7版本：Linux version 3.10.0-693.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017
涉及到以下工具的使用：
arm交叉編譯工具：arm-none-linux-gnueabi-gcc version 4.8.3
QEMU：qemu-2.11.0
uboot：u-boot-2018.01
Linux內核：linux-4.1.38
busybox：busybox-1.24.1
3 arm交叉工具安裝
我下載的版本是：arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu，直接解壓縮之后就可以使用，不需要安裝。
下載地址：
arm-none-linux-gnueabi-gcc 下載地址 - 代碼天地 (codetd.com)
安裝步驟：
1、解壓縮：
tar -vxf arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
我是放在了/usr/local/arm路徑下。
2、編輯vim /etc/profile
在文件末尾加上：
export PATH=$PATH:/usr/local/arm/arm-2014.05/bin
注意，上面的等號“=”是英文的，而不是中文的，否則會出現以下的錯誤。
-bash: export: `PATH＝/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/arm/arm-2014-05/bin': not a valid identifier
3、執行
source /etc/profile
4、測試
查看版本信息，這里遇到了一些問題，記錄如下：
$ arm-none-linux-gnueabi-gcc -v
-bash: /usr/local/arm/arm-2014-05/bin/arm-none-linux-gnueabi-gcc: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
提示少了so格式不對，根據網友說法，安裝glibc.i686之后解決了：
$ sudo yum install glibc.i686
查看版本信息：
$ arm-none-linux-gnueabi-gcc -v
... ...
Thread model: posix
gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29)
可以看出，安裝的版本是：gcc version 4.8.3 20140320
4 安裝QEMU
下載地址：
Index of / (qemu.org)，我下載的是2.11.0版本。
安裝步驟：
1、解壓縮：
tar -vxf qemu-2.11.0.tar.xz
2、配置編譯選項：
./configure --enable-kvm --enable-debug --enable-vnc --enable-werror --target-list=arm-softmmu --audio-drv-list=
configure腳本用於生成Makefile，其選項可以用./configure --help查看，這里使用到的選項含義如下：
--enable-kvm：編譯KVM模塊，使QEMU可以利用KVM來訪問硬件提供的虛擬化服務。
--enable-vnc：啟用VNC。
--enalbe-werror：編譯時，將所有的警告當作錯誤處理。
--target-list：選擇目標機器的架構。默認是將所有的架構都編譯，但為了更快的完成編譯，指定需要的架構即可。
這一步，出現了一些錯誤：
$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror --target-list=arm-softmmu --audio-drv-list=
ERROR: zlib check failed
Make sure to have the zlib libs and headers installed.
缺少zlib開發包，需要安裝好對應的版本：
$ yum install ghc-zlib-devel.x86_64
重新執行配置，報錯：
$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror --target-list=arm-softmmu --audio-drv-list=
ERROR: glib-2.22 gthread-2.0 is required to compile QEMU
需要安裝glib-2.22庫：
$ yum install glib2-devel.x86_64
重新執行配置，報錯：
$ ./configure --enable-kvm --enable-debug --enable-vnc --enable-werror --target-list=arm-softmmu --audio-drv-list=
ERROR: pixman >= 0.21.8 not present.
Please install the pixman devel package.
需要安裝pixman：
$ yum install pixman-devel.x86_64
再次執行，終於成功了。
3、編譯
$ make -j8
4、安裝
$ make install
... ...
install -d -m 0755 "/usr/local/bin"
install -c -m 0755 qemu-system-arm "/usr/local/bin"
5 安裝uboot
下載地址：
Index of /pub/u-boot/ (denx.de)
我下載的是：u-boot-2018.01.tar.bz2
安裝步驟：
1、解壓縮：
tar jvxf u-boot-2018.01.tar.bz2
2、進入uboot源文件目錄，執行：
export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
make vexpress_ca9x4_defconfig
make
3、會提示gcc版本過低：
$ make
scripts/kconfig/conf --silentoldconfig Kconfig
CHK include/config.h
CFG u-boot.cfg
GEN include/autoconf.mk
GEN include/autoconf.mk.dep
*** Your GCC is older than 6.0 and is not supported
可以選擇一個高版本的GCC重新安裝，或者修改uboot源文件的配置。
這里選擇修改uboot源文件的配置：
$ vim arch/arm/config.mk
注釋掉了65行。
4、編譯
直接執行make：
$ make
編譯完成后，如果目錄下生成 u-boot 文件，則說明編譯成功。
5、在U-Boot源碼目錄下編寫腳本 run.sh
qemu-system-arm \
-M vexpress-a9 \
-nographic \
-m 512M \
-kernel u-boot
然后 chmod +x run.sh 增加文件執行權限。
6、最后執行 ./run.sh
可以看到啟動了bootloader，但是會提示沒有映像文件。
Wrong Image Format for bootm command
ERROR: can't get kernel image!
6 編譯內核
下載地址：
Index of /pub/linux/kernel/v4.x/
我下載的是linux-4.1.38。
編譯步驟：
1、從arch/arm/configs下找到需要編譯內核的配置文件vexpress_defconfig，復制到源碼根目錄下。
$ cp arch/arm/configs/vexpress_defconfig ./
2、執行命令
export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
make vexpress_defconfig
make zImage
make modules
make dtbs
3、編譯后生成 在 arch/arm/boot 目錄下生成 zImage 文件，則說明編譯成功。
$ ls arch/arm/boot/
bootp/ compressed/ dts/ Image* install.sh Makefile zImage*
這個時候可以直接用QEMU來執行測試一下內核是否正常：
$ qemu-system-arm -M vexpress-a9 -m 512M -kernel ./linux-4.1.38/arch/arm/boot/zImage -dtb ./linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "console=ttyAMA0"
這里簡單介紹下qemu命令的參數：
-M vexpress-a9 模擬vexpress-a9單板，你可以使用-M ?參數來獲取該qemu版本支持的所有單板
-m 512M 單板運行物理內存512M
-kernel ./linux-4.1.38/arch/arm/boot/zImage 告訴qemu單板運行內核鏡像路徑
-nographic 不使用圖形化界面，只使用串口
-append "console=ttyAMA0" 內核啟動參數，這里告訴內核vexpress單板運行，串口設備是那個tty。
看到內核的打印，說明內核正常，但是這時候沒有文件系統，內核起不來。
此時只能通過殺死qemu-system-arm這個進程來退出。
ps -A | grep qemu-system-arm | awk '{print $1}' | xargs sudo kill -9
7 編譯busybox
下載地址：
Index of /downloads (busybox.net)
我下載的是：busybox-1.24.1.tar.bz2。
操作步驟：
1、進入 Busybox 源文件目錄下執行 make menuconfig。
$ cd busybox-1.24.1/
$ make menuconfig
選擇上Build BusyBox as a static binary (no shared libs)。
2、執行make和make install
$ make
LINK busybox_unstripped
Static linking against glibc, can't use --gc-sections
Trying libraries: crypt m
Library crypt is not needed, excluding it
Library m is needed, can't exclude it (yet)
Final link with: m
DOC busybox.pod
DOC BusyBox.txt
DOC busybox.1
DOC BusyBox.html
$ make install
--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------
_install 目錄下生成 需要的文件 bin linuxrc sbin usr 。
$ ls _install/
bin/ linuxrc@ sbin/ usr/
該目錄下的程序就是單板運行所需要的命令。
8 制作根文件系統
制作的根文件系統使用的是ext3文件格式。
先在Centos7主機環境下，形成目錄結構，里面存放的文件和目錄與單板上運行所需要的目錄結構完全一樣，然后再打包成鏡像（在開發板看來就是SD卡），這個臨時的目錄結構稱為根目錄。
操作步驟：
1、創建rootfs目錄（作為根目錄），根文件系統內的文件全部放到這里。
sudo mkdir -p rootfs/proc/
sudo mkdir -p rootfs/sys/
sudo mkdir -p rootfs/tmp/
sudo mkdir -p rootfs/root/
sudo mkdir -p rootfs/var/
sudo mkdir -p rootfs/mnt/
sudo cp etc rootfs/ -arf
etc目錄的下載地址：https://files.cnblogs.com/files/pengdonglin137/etc.tar.gz
2、拷貝busybox命令到根目錄下
cp ../qemu/busybox/busybox-1.24.1/_install/* -r ./
3、從工具鏈中拷貝運行庫到lib目錄下
cp -arf /usr/local/arm/arm-2014-05/arm-none-linux-gnueabi/lib rootfs/
rm rootfs/lib/*.a
arm-none-linux-gnueabi-strip rootfs/lib/*
4、創建4個tty端終設備
sudo mknod rootfs/dev/tty1 c 4 1
sudo mknod rootfs/dev/tty2 c 4 2
sudo mknod rootfs/dev/tty3 c 4 3
sudo mknod rootfs/dev/tty4 c 4 4
5、生成32M大小的鏡像
dd if=/dev/zero of=a9rootfs.ext3 bs=1M count=32
6、格式化成ext3文件系統
mkfs.ext3 a9rootfs.ext3
7、將文件拷貝到鏡像中
mkdir tmpfs
mount -t ext3 a9rootfs.ext3 tmpfs/ -o loop
cp -r rootfs/* tmpfs/
umount tmpfs
8、運行
qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/qemu/kernel/linux-4.1.38/arch/arm/boot/zImage -dtb /home/qemu/kernel/linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "root=/dev/mmcblk0 console=ttyAMA0" -sd a9rootfs.ext3
完整的啟動日志：
$ qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/qemu/kernel/linux-4.1.38/arch/arm/boot/zImage -dtb /home/qemu/kernel/linux-4.1.38/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "root=/dev/mmcblk0 rw console=ttyAMA0" -sd a9rootfs.ext3
WARNING: Image format was not specified for 'a9rootfs.ext3' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpuset
Linux version 4.1.38 (root@localhost.localdomain) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #1 SMP Sun Dec 26 16:34:51 CST 2021
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine model: V2P-CA9
Memory policy: Data cache writeback
CPU: All CPU(s) started in SVC mode.
PERCPU: Embedded 11 pages/cpu @9fbc0000 s16192 r8192 d20672 u45056
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048
Kernel command line: root=/dev/mmcblk0 rw console=ttyAMA0
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 12288 bytes
log_buf_len min size: 16384 bytes
log_buf_len: 32768 bytes
early log buf free: 14916(91%)
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 512652K/524288K available (4849K kernel code, 154K rwdata, 1376K rodata, 260K init, 148K bss, 11636K reserved, 0K cma-reserved)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xa0800000 - 0xff000000 (1512 MB)
lowmem : 0x80000000 - 0xa0000000 ( 512 MB)
modules : 0x7f000000 - 0x80000000 ( 16 MB)
.text : 0x80008000 - 0x8061c6a8 (6226 kB)
.init : 0x8061d000 - 0x8065e000 ( 260 kB)
.data : 0x8065e000 - 0x80684a00 ( 155 kB)
.bss : 0x80687000 - 0x806ac1f8 ( 149 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
Additional per-CPU info printed with stalls.
RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS:16 nr_irqs:16 16
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 dynamic clock gating disabled, standby mode disabled
L2C-310 cache controller enabled, 8 ways, 128 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x46420001
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
clocksource arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
Console: colour dummy device 80x30
Calibrating delay loop... 400.58 BogoMIPS (lpj=2002944)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x60008280 - 0x600082d8
Brought up 1 CPUs
SMP: Total of 1 processors activated (400.58 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 0
clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
cpuidle: using governor menu
of_amba_device_create(): amba_device_add() failed (-19) for /memory-controller@100e0000
of_amba_device_create(): amba_device_add() failed (-19) for /memory-controller@100e1000
of_amba_device_create(): amba_device_add() failed (-19) for /watchdog@100e5000
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@7,00000000/sysctl@01000
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@7,00000000/wdt@0f000
hw-breakpoint: debug architecture 0x4 unsupported.
Serial: AMBA PL011 UART driver
10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 38, base_baud = 0) is a PL011 rev1
console [ttyAMA0] enabled
1000a000.uart: ttyAMA1 at MMIO 0x1000a000 (irq = 39, base_baud = 0) is a PL011 rev1
1000b000.uart: ttyAMA2 at MMIO 0x1000b000 (irq = 40, base_baud = 0) is a PL011 rev1
1000c000.uart: ttyAMA3 at MMIO 0x1000c000 (irq = 41, base_baud = 0) is a PL011 rev1
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
Switched to clocksource arm,sp804
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 2, 16384 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 1 counters available
futex hash table entries: 1024 (order: 4, 65536 bytes)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
9p: Installing v9fs 9p2000 file system support
io scheduler noop registered (default)
clcd-pl11x 10020000.clcd: PL111 rev2 at 0x10020000
clcd-pl11x 10020000.clcd: /clcd@10020000 hardware, 1024x768@59 display
Console: switching to colour frame buffer device 128x48
clcd-pl11x 1001f000.clcd: PL111 rev2 at 0x1001f000
clcd-pl11x 1001f000.clcd: /smb/motherboard/iofpga@7,00000000/clcd@1f000 hardware, 640x480@59 display
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
Concatenating MTD devices:
(0): "40000000.flash"
(1): "40000000.flash"
into device "40000000.flash"
libphy: smsc911x-mdio: probed
smsc911x 4e000000.ethernet eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=4e000000.etherne:01, irq=-1)
smsc911x 4e000000.ethernet eth0: MAC Address: 52:54:00:12:34:56
isp1760 4f000000.usb: bus width: 32, oc: digital
isp1760 4f000000.usb: NXP ISP1760 USB Host Controller
isp1760 4f000000.usb: new USB bus registered, assigned bus number 1
isp1760 4f000000.usb: Scratch test failed.
isp1760 4f000000.usb: can't setup: -19
isp1760 4f000000.usb: USB bus 1 deregistered
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
rtc-pl031 10017000.rtc: rtc core: registered pl031 as rtc0
mmci-pl18x 10005000.mmci: Got CD GPIO
mmci-pl18x 10005000.mmci: Got WP GPIO
mmci-pl18x 10005000.mmci: No vqmmc regulator found
mmci-pl18x 10005000.mmci: mmc0: PL181 manf 41 rev0 at 0x10005000 irq 34,35 (pio)
input: AT Raw Set 2 keyboard as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10006000.kmi/serio0/input/input0
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
mmc0: new SD card at address 4567
mmcblk0: mmc0:4567 QEMU! 32.0 MiB
aaci-pl041 10004000.aaci: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
aaci-pl041 10004000.aaci: FIFO 512 entries
oprofile: using arm/armv7-ca9
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Registering SWP/SWPB emulation handler
rtc-pl031 10017000.rtc: setting system clock to 2021-12-26 13:20:20 UTC (1640524820)
ALSA device list:
#0: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10007000.kmi/serio1/input/input2
kjournald starting. Commit interval 5 seconds
EXT3-fs (mmcblk0): using internal journal
EXT3-fs (mmcblk0): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) on device 179:0.
Freeing unused kernel memory: 260K (8061d000 - 8065e000)
random: nonblocking pool is initialized
Please press Enter to activate this console.