Fast VM with crosvm

crosvm can be used to run a blazing fast Android VM on a Linux host with GPU acceleration, negligible overhead and close to native performance.
The Android window feels smoother than ever before.
It can configured to use udmabuf, minigbm and wayland for a incredibly efficient rendering pipeline with zero copy display from guest to host wayland server.
It doesn’t work as expected out of the box, so we will have to apply some patches to fix mouse input.

Follow the instructions on https://crosvm.dev/book/building_crosvm/linux.html to set up the environment.
Then switch to last known working commit for the patches:

git checkout c6ee39ce8

Apply patch to fix mouse input

curl https://xtr126.github.io/XtMapper-docs/00-crosvm-mouse-android.patch | patch -p1

Build crosvm with virgl support enabled

CROSVM_USE_SYSTEM_VIRGLRENDERER=1 cargo build --features gpu,virgl_renderer,wl-dmabuf --release

If minigbm build is giving errors, patch the Makefile as below. source

 sed 's/-Wall/-Wno-maybe-uninitialized/g' -i third_party/minigbm/Makefile

The executable will be in ./target/release/crosvm
Remove the CROSVM_USE_SYSTEM_VIRGLRENDERER=1 flags if build fails or you are using an ancient linux distro with older versions of libgbm or virglrenderer.

./target/release/crosvm run --cpus 2 --mem 2842 --disable-sandbox \
                --initrd ~/bliss-16.8/initrd.img \
                --params "root=/dev/ram0 quiet SRC=/ GRALLOC=minigbm_gbm_mesa HWC=drm_minigbm_celadon" \
                --rwdisk ~/bliss-16.8/android.img \
                --gpu=backend=virglrenderer,vulkan=false,udmabuf=true \
                --gpu-display=mode=windowed[1280,720],dpi=[160,160],refresh-rate=60 \
                --display-window-keyboard \
                --display-window-mouse \
                --wayland-sock=$XDG_RUNTIME_DIR/$WAYLAND_DISPLAY \
                ~/bliss-16.8/kernel

It cannot render to screen without using the right gralloc/hwc options. The android.img was prepared using the script.
Add SETUPWIZARD=0 to kernel parameters (—params ”..”) to bypass Android setup wizard.

Networking

https://crosvm.dev/book/running_crosvm/example_usage.html#add-networking-support

./tools/examples/setup_network

Add --net tap-name=crosvm_tap to the crosvm cmdline.

crosvm run
...
--net tap-name=crosvm_tap \
...

A static IP configuration is required inside the Android VM to connect to the network. However Android does not have settings to configure static IP for ethernet connections. We can use VirtWifi feature of Android-x86 as a workaround.
Add VIRT_WIFI=1 to kernel cmdline parameters to enable this feature.

--params "... VIRT_WIFI=1"

Then connect to VirtWifi network from Android settings. Select Static IP and set IP Address to 192.168.10.2 and gateway to 192.168.10.1

Troubleshooting

Colors are inverted: Turn on invert colors in Android Settings > Accessibility as a workaround or compile minigbm and virglrenderer with minigbm allocation enabled and link with crosvm binary.
crosvm crashes with virtio-gpu error: Remove udmabuf=true from crosvm cmdline, udmabuf is not supported by all GPUs.

Venus Vulkan driver

After applying the following patch, the embedded virglrenderer library is compiled with minigbm allocation. This is required on the host for venus.

curl https://xtr126.github.io/XtMapper-docs/01-crosvm-build-virglrenderer-minigbm-allocation.patch | patch -p1

Patching virglrenderer source is also required to link against libminigbm built by cargo.

cd third_party/virglrenderer
curl https://xtr126.github.io/XtMapper-docs/02-virglrenderer-link-crosvm-minigbm.patch | patch -p1
cd ../../

Setting the appropriate CFLAGS for your GPU allows minigbm to use native DRI backend, which also fixes the color issue. However it may not work on all GPUs.

CFLAGS=-DDRV_<I915 or AMDGPU> cargo build --features gpu,virgl_renderer,wl-dmabuf --release

Find virgl_render_server in build directory

find target/release/build/ -name virgl_render_server

Include vulkan=true in the list of comma separated gpu flags in crosvm cmdline and specify path to virgl_render_server found earlier.

...
--gpu=vulkan=true,udmabuf=true \
--gpu-display=mode=windowed[1280,720],dpi=[160,160],refresh-rate=60 \
--gpu-render-server path=/path/to/virgl_render_server  \
...

Enable serial console

Add androidboot.enable_console=1 to kernel cmdline parameters.

--params "... androidboot.enable_console=1"

For reference

crosvm command lines extracted from certain versions of Google products.

/usr/bin/crosvm --syslog-tag ARCVM(36) run --cpus 8 --mem 6713 --balloon-bias-mib 48 --net tap-fd=27 --net tap-fd=28 --net tap-fd=29 --net tap-fd=30 --net tap-fd=31 --net tap-fd=32 --cid 36 --socket /run/vm/vm.Sl5uEL/arcvm.sock --wayland-sock /run/chrome/wayland-0 --wayland-sock /run/arcvm/mojo/mojo-proxy.sock,name=mojo --no-smt --delay-rt --per-vm-core-scheduling --params root=/dev/vda init=/init androidboot.hardware=bertha androidboot.container=1 androidboot.host_is_in_vm=0 androidboot.dev_mode=1 androidboot.disable_runas=0 androidboot.chromeos_channel=stable androidboot.seneschal_server_port=16388 androidboot.iioservice_present=1 androidboot.arc_custom_tabs=0 androidboot.arc_file_picker=1 androidboot.enable_notifications_refresh=1 androidboot.lcd_density=160 androidboot.arc.primary_display_rotation=ORIENTATION_0 softlockup_panic=0 androidboot.enable_consumer_auto_update_toggle=1 androidboot.enable_privacy_hub_for_chrome=0 androidboot.keyboard_shortcut_helper_integration=0 androidboot.arcvm_virtio_blk_data=0 androidboot.zram_size=0 androidboot.arc_switch_to_keymint=0 androidboot.arcvm_mglru_reclaim_interval=30000 androidboot.arcvm_mglru_reclaim_swappiness=0 androidboot.arcvm_metrics_mem_psi_period=10 androidboot.arcvm_ureadahead_mode=readahead androidboot.native_bridge=libhoudini.so androidboot.arc.tts.caching=1 androidboot.arc_dalvik_memory_profile=8G androidboot.lmkd.vsock_timeout=100 androidboot.audio.aaudio_mmap_enabled=1 androidboot.audio.aaudio_mmap_period_size=512 androidboot.camera.async_process_capture_request=true ramoops.record_size=262144 ramoops.console_size=262144 ramoops.ftrace_size=4096 ramoops.pmsg_size=4096 ramoops.dump_oops=1 squashfs.cached_blks=20 --virtio-snd capture=true,backend=cras,client_type=arcvm,socket_type=unified,num_input_devices=3,num_output_devices=4,output_device_config=[[],[],[],[stream_type=pro_audio]],input_device_config=[[],[],[stream_type=pro_audio]] --disk /proc/self/fd/22,o_direct=false,block_size=4096 --disk /proc/self/fd/23,o_direct=false,block_size=4096 --disk /proc/self/fd/21,o_direct=false --disk /proc/self/fd/24,o_direct=false --disk /proc/self/fd/25,o_direct=false --gpu vulkan=true --gpu-render-server path=/usr/libexec/virgl_render_server --video-decoder libvda --video-encoder libvda --battery type=goldfish --shared-dir /run/arcvm/android-data:_data:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true:privileged_quota_uids=0 --shared-dir /run/arcvm/android-data:_data_media:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:ascii_casefold=true:writeback=true:privileged_quota_uids=0 --shared-dir /run/arcvm/media:stub:type=fs:cache=auto:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=1:rewrite-security-xattrs=true:ascii_casefold=true:writeback=false:posix_acl=false:privileged_quota_uids=0 --shared-dir /run/arcvm/host_generated/oem/etc:oem_etc:type=fs:cache=always:uidmap=0 299 1, 5000 600 50:gidmap=0 299 1, 5000 600 50:timeout=3600:rewrite-security-xattrs=true:writeback=true:posix_acl=false --shared-dir /run/arcvm/testharness:testharness:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /run/arcvm/apkcache:apkcache:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /usr/share/fonts:fonts:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /lib64:lib:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /usr/lib64:usr_lib:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /sbin:sbin:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /usr/bin:usr_bin:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --shared-dir /run/arcvm/jemalloc:jemalloc:type=fs:cache=always:uidmap=0 655360 5000,5000 600 50,5050 660410 1994950:gidmap=0 655360 1065,1065 20119 1,1066 656426 3934,5000 600 50,5050 660410 1994950:timeout=3600:rewrite-security-xattrs=true:writeback=true --vcpu-cgroup-path /sys/fs/cgroup/cpu/arcvm-vcpus --pstore path=/run/daemon-store/crosvm/97c77222b7089251463a24210e4e36bf5388530c/YXJjdm0=.pstore,size=1048576 --android-fstab /run/arcvm/host_generated/fstab --hugepages /opt/google/vms/android/vmlinux

/cuttlefish/bin/crosvm --extended-status run --socket=/tmp/cf_avd_0/cvd-1/internal/crosvm_control.sock --no-smt --no-usb --core-scheduling=false --vhost-user-mac80211-hwsim=/tmp/cf_env_0/env-1/vhost_user_mac80211 --gpu=backend=virglrenderer,context-types=virgl2,udmabuf=true,egl=true,surfaceless=true,glx=false,gles=true --gpu-display=mode=windowed[1280,720],dpi=[160,160],refresh-rate=60 --wayland-sock=/tmp/cf_avd_0/cvd-1/internal/frames.sock --rw-pmem-device=/cuttlefish/cuttlefish/instances/cvd-1/hwcomposer-pmem --mem=3096 --cpus=2 --rwdisk=/cuttlefish/cuttlefish/instances/cvd-1/os_composite.img --rwdisk=/cuttlefish/cuttlefish/instances/cvd-1/persistent_composite.img --multi-touch=/tmp/cf_avd_0/cvd-1/internal/touch_0.sock:1280:720 --rotary=/cuttlefish/cuttlefish/instances/cvd-1/internal/rotary.sock --keyboard=/tmp/cf_avd_0/cvd-1/internal/keyboard.sock --switches=/tmp/cf_avd_0/cvd-1/internal/switches.sock --net=tap-fd=118,mac="00:1a:11:e0:cf:00" --net=tap-fd=120,mac="00:1a:11:e1:cf:00" --rw-pmem-device=/cuttlefish/cuttlefish/instances/cvd-1/access-kregistry --pstore=path=/cuttlefish/cuttlefish/instances/cvd-1/pstore,size=2097152 --disable-sandbox --cid=3 --serial=hardware=virtio-console,num=1,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,console=true --serial=hardware=serial,num=1,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/kernel-log-pipe,earlycon=true --serial=hardware=virtio-console,num=2,type=sink --serial=hardware=virtio-console,num=3,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/logcat-pipe --serial=hardware=virtio-console,num=4,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/keymaster_fifo_vm.in --serial=hardware=virtio-console,num=5,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/gatekeeper_fifo_vm.in --serial=hardware=virtio-console,num=6,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/bt_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/bt_fifo_vm.in --serial=hardware=virtio-console,num=7,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/gnsshvc_fifo_vm.in --serial=hardware=virtio-console,num=8,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/locationhvc_fifo_vm.in --serial=hardware=virtio-console,num=9,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/confui_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/confui_fifo_vm.in --serial=hardware=virtio-console,num=10,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/uwb_fifo_vm.in --serial=hardware=virtio-console,num=11,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/oemlock_fifo_vm.in --serial=hardware=virtio-console,num=12,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/keymint_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/keymint_fifo_vm.in --serial=hardware=virtio-console,num=13,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/nfc_fifo_vm.in --serial=hardware=virtio-console,num=14,type=file,path=/cuttlefish/cuttlefish/instances/cvd-1/internal/sensors_fifo_vm.out,input=/cuttlefish/cuttlefish/instances/cvd-1/internal/sensors_fifo_vm.in --serial=hardware=virtio-console,num=15,type=sink --sound=/tmp/cf_avd_0/cvd-1/internal/audio_server.sock --bios=/cuttlefish/bootloader

crosvm.exe run-mp --bios=..\emulator\avd\bios.rom --crash-pipe-name=\.\pipe\crashpad_9072_JBFCMAATNYKRGSDL --cpus=6 --mem=6144 --init-mem=3072 --flex-mem-chunk-size=64 --log-file="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\Emulator.log" --kernel-log-file="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\Hypervisor.log" --logs-directory="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\emulator_logs" --serial=hardware=serial,num=1,type=file,path="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\AndroidSerial.log",earlycon=true --serial=hardware=virtio-console,num=1,type=file,path="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\AndroidSerial.log",console=true --pstore=path="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\pstore",size=1048576 --disk="D:\Program Files\Google\Play Games Developer Emulator\23.11.1397.6\emulator\avd\aggregate.img",sparse=false,io_concurrency=32,async_executor=overlapped --rwdisk="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\userdata_vp505wie.ntt\avd\metadata.img",sparse=true,io_concurrency=32,async_executor=overlapped --rwdisk="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\userdata_vp505wie.ntt\avd\userdata.img",sparse=true,io_concurrency=32,async_executor=overlapped --rwdisk="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\userdata_vp505wie.ntt\avd\misc.img",sparse=true,io_concurrency=32,async_executor=overlapped --process-invariants-handle=5384 --process-invariants-size=2959 --gpu=angle=true,backend=gfxstream,egl=true,gles=false,glx=false,surfaceless=false,vulkan=true,wsi=vk,vk_mem_file="C:\Users\x\AppData\Local\Google\Play Games Developer Emulator\Logs\emulator_logs\vk_abort_mem_info.log",gpu_uuid=[134,128,23,89,7,0,0,0,0,0,0,0,0,0,0,0],driver_uuid=[51,49,46,48,46,49,48,49,46,50,49,49,49,0,0,0],displays=[[mode=windowed[1920,1080],refresh-rate=30,hidden]] --spu=gatekeeper_port=31540,keymaster_port=31541 --host-guid=dc55608a-e3e6-43cc-ab85-156b69d98b9d --ime-port=60006 --enable-audio-capture --params="androidboot.boot_devices=pci0000:00/0000:00:01.0,pci0000:00/0000:00:02.0,pci0000:00/0000:00:03.0,pci0000:00/0000:00:04.0 androidboot.console=hvc0 androidboot.hardware.gltransport=virtio-gpu-asg androidboot.hardware.gralloc=minigbm androidboot.hardware.hwcomposer=ranchu androidboot.kiwi.adbproxy.enabled=1 androidboot.kiwi.adbproxy.port=38068 androidboot.kiwi.client_version=23.11.1397.6 androidboot.kiwi.logcatserial=1 androidboot.kiwi.logcatserial.dev=/dev/hvc0 androidboot.kiwi.logcatserial.filter=System.err:E,.android.common:E,*:I androidboot.kiwi.metrics_group=unknown androidboot.partition_map=vdb,metadata;vdc,userdata;vdd,misc androidboot.serialno=420c4e54eb7349a1ae83a486e31bc6be androidboot.pcs.recvport=50051 androidboot.pcs.sendport=50050 androidboot.timezone= androidboot.hw_timeout_multiplier=1 noxsaves tsc=reliable androidboot.pcs.use_vsock=true androidboot.pcs.enable_host_ime=true androidboot.ime.ime_port=60006 androidboot.pcs.enable_volume_control=true androidboot.pcs.enable_pre_game_launch_memory_compaction=true androidboot.opengles.version=196609 androidboot.dirty.expire.centisecs=500 androidboot.kiwi.prefetch.io_depth=20 androidboot.kiwi.prefetch.max_fds=2000 androidboot.kiwi.prefetch=replay androidboot.async.persist.writes=true androidboot.enable.min.free.kbytes=1 androidboot.standard.min.free.kbytes=61440 androidboot.optimized.min.free.kbytes=30720 snd-hda-intel.enable=0 snd_intel8x0.inside_vm=1 snd_intel8x0.ac97_clock=48000 androidboot.asg.backoffiters=400000 androidboot.asg.backoffincrement=1000 androidboot.hardware.egl=angle androidboot.hardware.vulkan=ranchu nopat androidboot.kiwi.enable_houdini=1 androidboot.locale=en-US androidboot.vsock_gatekeeper_port=31540 androidboot.vsock_keymaster_port=31541 androidboot.kiwi_out_period_count=8 virtio_snd.msg_timeout_ms=5000 androidboot.pcs.events_service=true androidboot.angle_log_texture_decompression_events=true" --ac97 backend=win_audio --cid=3 --multi-touch=nil --mouse=nil --kiwi-mouse=nil --product-version="23.11.1397.6" --product-channel="DevUnknown" --product-name="Play Games Developer Emulator" --log-source="0" --service-pipe-name="service-ipc-a5168ce3-5b2a-431e-a93b-89f1f9c5cb0e" --pvclock  crosvm.exe