Windows10 guest support by nevilad · Pull Request #204 · intel/haxm (original) (raw)

Saving cr_pat/cr8 to/from qemu:

set HAX_CUR_VERSION to 5

add cr8 to the end of vcpu structure received through ioctl

create patch for qemu which increments supported API version, sets/gets cr_pat with other MSRs and add cr8 to it's vcpu structure. When qemu sees haxm of an older version, it will not set cr_pat with other MSRs.

when hax works with an older qemu version (no cr8 and cr_pat emulation), it will not save/restore them.
Thus new functionality will work only, when qemu is aware of them.

Excellent! If you changed some common CPU code in qemu, then there could be more comments from the community. Please check this link for more information:
https://wiki.qemu.org/Documentation/GettingStartedDevelopers

There was a comment "pat is disabled!"
Still no answer.

I can't find the history why the author left such comment. We could run more tests based on you PAT enhancement, if it runs OK for both upstream QEMU and Android Emulator, it could be safe then.

I can't yet find a solution to the CR8 problem.
Windows 10 x64 installation enters an eternal cycle. Don't know what it wants, but previous errors where due to missed CPU functionality. Seems that the problem is not in performance due to cr8 load\store vmexits.

If the "nonstandard solution" you mentioned is the case that HAXM handle APIC by disable rwx in page, the reason could be we doesn't support APICv like KVM yet.
What is the specific "CR8 problem" you met in this PR? The performance is too bad, or simply handle CR8 doesn't work at all for 64bit Windows? As @krytarowski mentioned, CR8 is passed to qemu and emulate by qemu. If it's the performance issue, than APICv has to be implemented.
If you disassemble the 64-bit nt kernel, you can see 64-bit Windows IRQL mechanism relies on CR8: (Dump from WinDbg with public symbol)

0: kd> uf nt!KeGetCurrentIrql  
nt!BgpFwGetCurrentIrql:  
fffff802`2b266620 440f20c0        mov     rax,cr8  
fffff802`2b266624 c3              ret

0: kd> uf nt!KzRaiseIrql
nt!KzRaiseIrql:  
fffff802`2b266600 440f20c0        mov     rax,cr8  
fffff802`2b266604 0fb6d1          movzx   edx,cl  
fffff802`2b266607 440f22c2        mov     cr8,rdx  
fffff802`2b26660b 8b15938f5000    mov     edx,dword ptr [nt!KiIrqlFlags (fffff802`2b76f5a4)]  
fffff802`2b266611 85d2            test    edx,edx  
fffff802`2b266613 0f85eb1f1700    jne     nt!KzRaiseIrql+0x172004 (fffff802`2b3d8604) Branch  
fffff802`2b266619 c3              ret  Branch  
fffff802`2b3d8604 f6c201          test    dl,1  
fffff802`2b3d8607 0f840ce0e8ff    je      nt!KzRaiseIrql+0x19 (fffff802`2b266619)  Branch  
fffff802`2b3d860d 80f902          cmp     cl,2  
fffff802`2b3d8610 0f8203e0e8ff    jb      nt!KzRaiseIrql+0x19 (fffff802`2b266619)  Branch  
fffff802`2b3d8616 3c02            cmp     al,2  
fffff802`2b3d8618 0f83fbdfe8ff    jae     nt!KzRaiseIrql+0x19 (fffff802`2b266619)  Branch  
fffff802`2b3d861e 65488b0c2520000000 mov   rcx,qword ptr gs:[20h]  
fffff802`2b3d8627 488b91b8610000  mov     rdx,qword ptr [rcx+61B8h]  
fffff802`2b3d862e f0810a00000100  lock or dword ptr [rdx],10000h  
fffff802`2b3d8635 c3              ret

0: kd> uf nt!KzLowerIrql
nt!KzLowerIrql:
fffff802`2b2d05f0 4053            push    rbx
fffff802`2b2d05f2 4883ec20        sub     rsp,20h
fffff802`2b2d05f6 8b05a8ef4900    mov     eax,dword ptr [nt!KiIrqlFlags (fffff802`2b76f5a4)]
fffff802`2b2d05fc 0fb6d9          movzx   ebx,cl
fffff802`2b2d05ff 85c0            test    eax,eax
fffff802`2b2d0601 0f85f1881000    jne     nt!KzLowerIrql+0x108908 (fffff802`2b3d8ef8)  Branch
fffff802`2b2d0607 0fb6c3          movzx   eax,bl
fffff802`2b2d060a 440f22c0        mov     cr8,rax
fffff802`2b2d060e 4883c420        add     rsp,20h
fffff802`2b2d0612 5b              pop     rbx
fffff802`2b2d0613 c3              ret
fffff802`2b3d8ef8 a801            test    al,1
fffff802`2b3d8efa 0f840777efff    je      nt!KzLowerIrql+0x17 (fffff802`2b2d0607)  Branch
fffff802`2b3d8f00 440f20c0        mov     rax,cr8
fffff802`2b3d8f04 3c02            cmp     al,2
fffff802`2b3d8f06 0f82fb76efff    jb      nt!KzLowerIrql+0x17 (fffff802`2b2d0607)  Branch
fffff802`2b3d8f0c 80fb02          cmp     bl,2
fffff802`2b3d8f0f 0f83f276efff    jae     nt!KzLowerIrql+0x17 (fffff802`2b2d0607)  Branch
fffff802`2b3d8f15 65488b0c2520000000 mov   rcx,qword ptr gs:[20h]
fffff802`2b3d8f1e 488b81b8610000  mov     rax,qword ptr [rcx+61B8h]
fffff802`2b3d8f25 f08120fffffeff  lock and dword ptr [rax],0FFFEFFFFh
fffff802`2b3d8f2c e8eb420d00      call    nt!KiRemoveSystemWorkPriorityKick (fffff802`2b4ad21c)
fffff802`2b3d8f31 90              nop
fffff802`2b3d8f32 e9d076efff      jmp     nt!KzLowerIrql+0x17 (fffff802`2b2d0607)  Branch

Thus any operations on IRQL (which are quite frequent) will cause tons of CR8 access. To handle CR8 correctly, certain IRQs have to be handled as well.