Let me give you a simple example. CPUID is a peculiar instruction; it's the only non ring-0 allowed instruction that will unconditionally trigger a VM-exit (Vol.3B:253669, p21-2). If you trace over CPUID in your favorite debugger, the VM-exit will happen: the CPU enters VMX-Root mode, and your VMM is called to handle the offending instruction.
The lazy coder would write something like:
// emulate CPUID
if(Reason == EXIT_REASON_CPUID) then
pushad
mov eax, GuestEax
mov ecx, GuestEcx
cpuid
mov GuestEax, eax
mov GuestEcx, ecx
mov GuestEdx, edx
mov GuestEbx, ebx
popad
...
// update isntruction pointer
GuestEip += 2
vmresume()
Well, that's not enough. Execution will resume after CPUID. The trap flag will raise the debug exception after executing the instruction that follows... and there you have one simple way to detect poorly coded HVMs. One way to prevent this would be to inject a vectored event (INT1) on VM-entry.
This piece of assembly code implements the above trick (if we may call it so):
call GetCurrentProcess
push 1
push eax
call SetProcessAffinityMask
push offset seh
push dword ptr fs:[0]
mov fs:[0], esp
pushfd
or dword ptr [esp], 100h
popfd
cpuid
traphere:
mov eax, 1
jmp done
seh:
mov eax, [esp+0Ch]
cmp dword ptr [eax+0B8h], traphere
setne al
movzx eax, al
done:
add eax, 30h
push eax
push esp
call crt_printf
push eax
call ExitProcess
(On SMP systems, change the thread affinity to execute the test on other CPUs.)
There are other similar ways to detect a hypervisor from user-mode - again, assuming its implementation is lacking.
