13:03 <kensan> mato: Hi, thanks for the review.

13:04 <kensan> mato: Regarding the issue of testing on Muen: what kind of setup do you have in mind? Do you have any (rough) ideas?

14:59 <mato> kensan: I'm not sure, I think it'd be nice to have some automated way of testing (even if just under Bochs) that things still run as well as just build.

15:00 <mato> kensan: However, I don't think we need to hold up the PR waiting for that to be implemented.

15:03 <mato> kensan: However, I am going to try and test it manually, just for my peace of mind that "things work" :-)

15:17 <kensan> mato: Cool, thanks!

15:18 <kensan> mato: Just poke me if you run into trouble.

15:18 <kensan> mato: Unfortunately the whole setup is a bit involved by the nature of Muen requiring Intel VT-x and a detailed hardware spec.

15:39 <kensan> mato: Ah btw, test_time also does not work in Bochs since time is not emulated faithfully.

15:41 <mato> kensan: I didn't realise Bochs emulated VT-x, some emulation is better than none. Will it eventually be possible to run under nested KVM? My understanding is recent kernels have improved in that regard.

15:43 <kensan> mato: Yes, Bochs has support for VT-x and it is quite good. Initially we found some small issues but they were fixed quite quickly. That is why the version must be recent enough so the fixes are included.

15:43 <kensan> mato: If nested virtualization is stable it should in general be possible to run under KVM.

15:49 <mato> kensan: supposedly Linux 4.10+ can now run stock Bhyve under nested KVM: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203994

15:49 <mato> kensan: And there are a bunch of KVM changes including nested virt in 4.11: http://lkml.iu.edu/hypermail/linux/kernel/1702.2/04145.html

15:50 <dstolfa> mato: FWIW, there's been some mumble to get some support for nested VT-x in bhyve too

15:50 <mato> Of course, I wonder how secure nested VT-x can be since the hypervisor has to do quite a lot more to support it.

15:50 <mato> However, for development/testing/CI it'll be awesome.

15:50 <dstolfa> mato: Well, depends on how you implement it of course

15:51 <dstolfa> It's largely used for development, yes

15:51 <dstolfa> Using nested VT-x in production is a little insane

15:51 <dstolfa> Considering you have no way of easily verifying it's correct, secure, and well, it performs like poop

15:51 <kensan> mato: Yeah, I would say complexity increase is by an order of magnitude

15:51 <dstolfa> kensan: In bhyve's case, maybe even more

15:51 <mato> Hmm. So you think we'll never see "production quality" nested VT-x?

15:52 <kensan> mato: even when using the hardware features.

15:52 <dstolfa> mato: I don't think that it's feasible to rely on it happening in the present, maybe sometime we will

15:52 <dstolfa> Depends on the CPU trends

15:52 <dstolfa> If we have to emulate everything in the hypervisor, I don't think it's feasible to run in a production environment at reasonable speed

15:52 <dstolfa> But if CPUs give some support for nested virtualization, I don't see why not

15:53 <kensan> mato: Well, I am not sure. People are happy with the current crop of hypervisors so throwing nested virtualization into the mix may not be such a big increase/issue...

15:53 <dstolfa> The problem isn't as much in merging N VMCS into one, but in emulating VMX

15:53 <dstolfa> kensan: Well, if speed isn't your primary concern, then you could run nested, yes

15:54 <dstolfa> But you'd have to expect failures in terms of hypervisor's emulation of VMX, which isn't that uncommon

15:55 <kensan> dstolfa: Yeah, that is what I meant with complexity increase: that will definitely bring some interesting bugs.

15:55 <kensan> Case in point: the recent nasty Xen bugs have been in emulation code...

15:56 <mato> Yeah, and my limited exposure to the actual hypervisor VMX code (and Intel manuals) certainly made my eyes glaze over in terms of the complexity involved.

15:57 <dstolfa> mato: kensan One thing that is more feasible to expect is running MirageOS inside containers. I think Joyent does something similar -- they spawn an illumos zone and spin up a VM inside that

15:57 <mato> Emulating all of it correctly and securely is a huge task.

15:57 <dstolfa> mato: The pseudocode of VMX is enormous

15:58 <dstolfa> Had lots of fun with that trying to figure things out :D

16:00 <kensan> dstolfa: Its funny that in some places the pseudo-code just turns into comments :) Almost like the Intel engineer writing the documentation was getting tired.

16:01 <dstolfa> kensan: Oh yeah, AMD's manual is like that too lol

16:01 <kensan> mato: Its the reason we use the bare minimum. Plus we restrict subjects/VMs to run on the same logical CPU all the time so no headaches regarding migration.

16:02 <dstolfa> I spent about 2 full days digging through Intel and AMD manuals trying to figure out how to disable VMCALL and VMMCALL, turns out Intel doesn't have a way for it lol

16:03 <kensan> dstolfa: Are you working on bhyve or why do you burden yourself with this? ;)

16:03 <dstolfa> kensan: Working on bhyve in the context of real time tracing of VMs

16:03 <dstolfa> Using bhyve as my reference hypervisor, and extending it along the way

16:04 <dstolfa> (But the idea is obviously to have it run seamlessly on any hypervisor or OS)

16:05 <kensan> dstolfa: cool.

16:05 <dstolfa> The context for my digging through manuals in terms of VMCALL/VMMCALL was when implementing a hypercall ABI for the fast trap mechanism from the guest kernel to the host kernel while tracing, but in doing so I also wanted to provide a way to do automated malware analysis in the future and went about making a framework around emulating other hypervisors in bhyve, which turned out to require a fair bit of

16:05 <dstolfa> digging

16:05 <dstolfa> Also I wanted to provide a way to enable or disable hypercalls

16:06 <kensan> dstolfa: Ah I see. imho you can always execute vmcall which traps, right?

16:06 <dstolfa> kensan: Yeah, you can't stop .byte 0x0f,0x01,0xc1 from going to VMX root mode, you can just stop it from doing anything meaningful and injecting an UD fault

16:07 <kensan> dstolfa: Yeah, but the hypervisor needs to always handle it... There's a bunch of instructions which trap unconditionally.

16:08 <dstolfa> kensan: Oh yeah, but AMD provides a way to disable access to VMMCALL on a VMCB level

16:08 <kensan> mato: Regarding the Signed-off-by: do you want me to add this to all commits?

16:08 <kensan> dstolfa: That's nice.

16:09 <mato> kensan: We don't currently have that as a policy for contributions, so no. I'll make a comment in the PR.

16:09 <kensan> mato: Ah ok. I updated the PR without making that change so I thought I'd ask.

16:42 <mato> kensan: Did you add some commits to the Muen branch since I reviewed it? Moving the call to console_init()? I can't quite tell from the Github web UI.

16:46 <mato> Ah, it seems you rebased it with that change. Not obvious from Github :-/

16:48 <kensan> mato: Yeah, I fixed up the original commit.

16:48 <kensan> mato: instead of doing a commit on top.

16:49 <kensan> mato: Hm so github shows all commits as changed since the hashes are different from the commit I reworked :/

16:52 <mato> Yeah.

16:53 <mato> Anyway, it looks fine. I'll test it tomorrow and then hit the green button if it all works.

16:53 <mato> GTG. TTYL.

16:56 <kensan> mato: cool thanks!

18:26 <hannes> yeah, nested virtualisation using software sounds a step backwards

18:27 <hannes> and I've some theory it is not needed.... in most cases (*working hard on some libvirt replacement to provision other peoples vms on my physical machine*)