Migrate a Centos 7 VM from VirtualBox to Hyper-V

Recently I decided I wanted to install Docker Desktop for Windows in order to do some local testing.  The installation process is pretty straightforward but requires Hyper-V be enabled on the machine.  This is automatically done by the installer, but contains one minor inconvenient caveat: once Hyper-V is enabled, VirtualBox no longer functions.  My understanding for this is that Hyper-V is a ring-1 hypervisor which runs your primary OS as an (albeit special) VM.  Because of that, it's no longer possible to run a ring-2 hypervisor like VirtualBox.

There's plenty of guides out there on how to make this migration, but they all seem to be missing at least a few important points.  For the most part I followed this guide which has links to all of the required tools.  Here's the list of issues that I faced that were not covered in that article:

1. Convert-VHD would not recognize by vmdk file.  Ultimately I ended up finding this article which proposed an alternate command: ConvertTo-MvmcVirtualHardDisk. This worked as described in the original article.

2. The original article is a little vague on which lines might need to be commented out in the "desc.txt" file that is created.  I ultimately ended up needing to comment out these:

#ddb.uuid.image="8ce66459-16da-4a18-a56b-6149a49e0de3"
#ddb.uuid.parent="00000000-0000-0000-0000-000000000000"
#ddb.uuid.modification="00000000-0000-0000-0000-000000000000"
#ddb.uuid.parentmodification="00000000-0000-0000-0000-000000000000"

3. Once those issues were resolved, I had a new vhdx that I was able to attach to a new Hyper-V VM.  Unfortunately it wouldn't boot.  On a normal boot the VM would hang at the graphical boot screen for a few minutes, then fail and dump to the dracut recovery shell.  A look at journalctl indicated that this was because none of the disks were found.  A lot of articles on this issue recommend rebuilding the initramfs by rebooting and choosing the "rescue" option on the GRUB boot menu.  I did find that the rescue option would boot, but unfortunately rebuilding initramfs with: dracut --force didn't seem to have any effect.  Luckily the nice folks on freenode IRC #centos pointed me in the right direction:  my virtualbox VM had the guest additions installed still!  This article explains how to remove the vbox guest additions, but basically just run the installer again with the uninstall argument:

sudo sh /media/VBOXADDITIONS_4.1.10_76795/VBoxLinuxAdditions.run uninstall

Obviously hyper-v has no menu item to mount the guest additions CD, but the .iso can be found at:

C:\Program Files\Oracle\VirtualBox\VBoxGuestAdditions.iso

Once that was uninstalled, one final reboot and... bob's your uncle!

Structure com.vmware.appliance.recovery.backup.job.details.info has a union with a field not required for this case = end_time

A VCSA 6.7 backup may fail with this error.  The most common explanation is that this is related to file permissions as described in this article.  But what if you're not using an IIS ftp server?  I was testing this using FileZilla FTP Server on Windows 10 1809.  In my case the service was running as SYSTEM, which already had full access to the FTP directory.  In fact it was creating subdirectories, but would fail when attempting to upload a file.

What I discovered was that the VCSA uses passive mode FTP with no option to change that fact.  By default passive mode servers choose random ports >1023 and pass them to the client which will then open a data connection to that port as described here. Unfortunately the Windows Firewall on this machine was set in the correct restrictive fashion, so those passive mode ports were never permitted.

It is possible, however, to choose custom passive mode ports in FileZilla.  In the server settings I chose to use the range 1024-1048, and then whitelisted those same ports in the firewall.  After that the backup completes successfully. 

Disable POP3 and IMAP on all mailboxes Office 365

I was recently annoyed to learn that there is no tenant wide way to disable POP3 or IMAP in Exchange Online.  Luckily PowerShell makes this task quite simple.  This article discusses the various commands that are available, but essentially you'll need to connect to Exchange Online via PowerShell and then run:

Get-User -ResultSize Unlimited | Set-CasMailbox -PopEnabled $false -ImapEnabled $false

Unfortunately you'll have to remember to disable these protocols for every new mailbox that is provisioned. This can also be performed from the ECP in the Mailbox Features.

Install vSphere 4 client on Windows 10

I ran into a number of errors attempting to install an older vSphere client in Windows 10. The first error says "This product can only be installed on Windows XP SP2 and above".  This issue is described in this VMware KB article.  I fixed this by right-clicking the installer executable and selecting to run it in Windows XP SP3 compatibility mode.  After that the installer seemed to run but would ultimately silently fail without installing anything.  It turns out that the installer requires .NET 3.5 to be installed.  These instructions from Microsoft show how to install this prerequisite in Windows 10, and it did not require a reboot for me.  Once vSphere 4 was installed I was able to successfully run it as well as update to later version when connecting to vCenter servers on newer ESXi.

Spiceworks HTTPS Redirect Breaks Stuff

We recently moved our Spiceworks installation to HTTPS.  While there's a handy setting in the options to force user connections to the portal to use HTTPS, this doesn't affect the backend used by admin and helpdesk staff.  There's a lot of bad advice out there about how to accomplish this redirect. Many threads like this one suggest adding a 302 redirect to a port 80 virtualhost to redirect to https.  While this does appear to work initially you will find that incoming emails no longer generate tickets.   If you view the production.txt log in C:\Program Files(x86)\Spiceworks\log\ you'll see an entry like this:

I[08:12:09.44 9b1030] scheduled call to check for ticket email url_ping: /tickets/check_email (http://127.0.0.1:80/tickets/check_email)
W[08:12:09.44 9b1030] check for ticket email url_ping: /tickets/check_email => unexpected response: Net::HTTPMovedPermanently

Yep, that's right - Spiceworks uses an internal API on port 80.  What's worse is that it does not follow the 302 redirect correctly, so if you go this route it will not work.  Luckily there are a number of other threads like this one on the Spiceworks forum that have a better suggestion - using Apache mod_rewrite to accomplish the task.  Specifically you will need to add the following directives to the httpd.conf file in C:\Program Files(x86)\Spiceworks\httpd\conf:

RewriteEngine On
RewriteCond %{REMOTE_HOST} !^127\.0\.0\.1
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

This will also require that you have the directive:

LoadModule rewrite_module modules/mod_rewrite.so

somewhere in the config but this appears to happen by default.  Essentially this will rewrite any HTTP requests to HTTPS with the exception of 127.0.0.1, the loopback addressed used by Spiceworks for internal API calls.

Hybrid Exchange Writeback Permissions

I recently ran into an issue after configuring Azure Active Directory Connect with hybrid Exchange that certain attributes couldn't be written back to the on-prem directory.  This manifests as errors in the sync tool, specifically a "Connected data source error code 8344" and "Insufficient access rights to perform the operation" on the export task. There's plenty of documentation that shows which permissions are required to support writeback of exactly 8 attributes.   For some reason it seems that the AAD Connect setup tool does not correctly add these permissions when selecting Hybrid Exchange mode.  There's a number of scripts out there, but two that I'll point out are this one from the Technet Gallery which appears to support a number of different configuration scenarios, as well as this one from c7solutions which is quite simple but effective.  The c7 post also has a great explanation of why these types of scripts are necessary.  The script is so useful that I've also generated an archive of the page here, in case it is ever moved/removed.

After running the script for a couple of minutes most export errors were resolved.  The specific issue can also be caused by an AD object with blocked inheritance.  This script from the technet gallery can be used to discover which users have inheritance blocked.  Once found they can either be fixed, or could be manually targeted for permissions with the aforementioned scripts.

Princeton Bitcoin Textbook

In case you hadn't heard Bitcoin hit at an all-time high of over $2,400 USD / BTC today.  There's plenty of good cursory information about Bitcoin but if you're looking for a decent in depth discussion of Bitcoin, related protocols, and other associated topics check out this book published by Princeton University Press.  The book is particularly suited to readers who already have an understanding of cryptography, computer science, networking etc.  If you're interested in going even deeper there's also a Coursera course to accompany it developed by one of the authors of the book.