Hybrid Exchange Writeback Permissions

I recently ran into an issue after configuring Azure Active Directory Connect with hybrid Exchange that certain attributes couldn't be written back to the on-prem directory.  This manifests as errors in the sync tool, specifically a "Connected data source error code 8344" and "Insufficient access rights to perform the operation" on the export task. There's plenty of documentation that shows which permissions are required to support writeback of exactly 8 attributes.   For some reason it seems that the AAD Connect setup tool does not correctly add these permissions when selecting Hybrid Exchange mode.  There's a number of scripts out there, but two that I'll point out are this one from the Technet Gallery which appears to support a number of different configuration scenarios, as well as this one from c7solutions which is quite simple but effective.  The c7 post also has a great explanation of why these types of scripts are necessary.  The script is so useful that I've also generated an archive of the page here, in case it is ever moved/removed.

After running the script for a couple of minutes most export errors were resolved.  The specific issue can also be caused by an AD object with blocked inheritance.  This script from the technet gallery can be used to discover which users have inheritance blocked.  Once found they can either be fixed, or could be manually targeted for permissions with the aforementioned scripts.

Princeton Bitcoin Textbook

In case you hadn't heard Bitcoin hit at an all-time high of over $2,400 USD / BTC today.  There's plenty of good cursory information about Bitcoin but if you're looking for a decent in depth discussion of Bitcoin, related protocols, and other associated topics check out this book published by Princeton University Press.  The book is particularly suited to readers who already have an understanding of cryptography, computer science, networking etc.  If you're interested in going even deeper there's also a Coursera course to accompany it developed by one of the authors of the book.

O365 Migration Endpoint Creation Error

When creating a migration endpoint you may receive the error that "No MRSProxy was found running at 'name.domain.com'" with the name of your email server from autodiscover.  If you check the EWS virtual directory you will see that in fact the MRSProxy is enabled.  Further, if you check the application event log on the Exchange server you will see Event ID 1309 from Source ASP.NET.  This was a very frustrating error as it prevented the creation of migration endpoints on either the Exchange on-prem or online side of the equation.  Luckily I came across this thread which explained that it's necessary to  recycle the MSExchangeServicesAppPool on the on-prem Exchange server.  This was a quick fix with no observable impact to users.  After performing this step migration endpoint creation was quick and painless.

Audit File System in Server 2012 R2 and Event 4656

When I recently enabled file system auditing on a Windows Server 2012 R2 I was overwhelmed by the volume of events generated.  I'm talking >4 GB/day in some instances!  When I manually inspected the security event logs it appears to me that the majority of the events generated were event id 4656.  According to all of the documentation I can find this event should only be logged if the Audit Handle Manipulation subcategory of Object Access auditing is enabled.  In my case I hadn't enabled it!  Even the official MS documentation doesn't mention that event.

When searching for info I came across this comment at the bottom of a serverfault post:

Currently, under Server 2012 R2 events 4656 will generate even if Handle Manipulation category is disabled. In our case, we have enabled Audit File System category which was only generating 4660-4663 events on previous Server versions (2008-2008R2-2012) but on Server 2012 R2 this initiates overwhelming flow of 4656 events. The issue has been reported to Microsoft however there is no resolution yet.

I've yet to come across any official discussion of this, but  this certainly corresponds with my experience.  This thread was from June 2016, with a followup comment in January 2017.  Is this an undocumented feature?  A bug?

Luckily for me I'm using Splunk to ingest these logs so I was simply able to add:

blacklist1=EventCode="4656"

in the [WinEventLog://Security] stanza of C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf in order to filter it out.  YMMV!

Set management IP on a VLAN on Dell Force10 S55 Switch Stack

While these switches do have dedicated management ethernet ports, it's often simpler and neater to set a management IP on the normal out of band / management VLAN that's already trunked on the uplink. For whatever reason the official Dell knowledge base article on the subject leaves out a crucial detail! It's necessary to set a default route before communication can commence. If you haven't set an IP on the management port you can't issue the management route command. Instead you need to issue the following:

ip route 0.0.0.0/0 192.168.0.1

to set a normal default gateway for the switch. Once that's done you're all set!

VMCA Intermediate CA

In vCenter 6 (and maybe some earlier versions) it's possible to configure the vCenter server to act as a subordinate CA to your existing PKI and issue all certs as part of a trusted chain. There's lots of instructions on the VMware knowledge base, but there are at least two critical errors in the CLI guide.

On this page that describes the process of compiling the VMCA Chained Cert that includes your root (and any other intermediate certs) it clearly shows the order as:

-----BEGIN CERTIFICATE-----
Certificate of VMCA
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Certificate of intermediary CA
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Certificate of Root CA
-----END CERTIFICATE-----

But if you try this you will receive errors regarding an invalid cert chain. Specifically you will see:

Error Code : 70063
Error Message : Invalid Certificate Chain was gives as input

That's because this is exactly backwards! The new cert needs to be at the top, followed by the intermediate, then the root. I learned that on a couple of different blogs. I was hesitant to believe it at first, but once I found it in multiple sources I gave it a shot and sure enough it worked.

The second error relates to the vpxd service not restarting in time, leading the certificate-manager to attempt to rollback the changes (which fails). This situation is described in this forum posting, but no answer is given. After many hours of testing, and digging through logs, etc, a possible solution was discovered on this only tangentially related forum thread: nowhere in the documentation is there any mention of required OU entries for any of the certificates, but this blog post states that as of 6.0u1b and 6.0u2 there are in fact distinct requirements. They are as follows

For the MACHINE CSR use "Root" for Organizational Unit (OU)

For User Solution User Certificate CSRs:
For Machine, use "Machine" as OU
For vsphere-webclient, use "WebClient" as OU
For vpxd, use "VPXD" as OU
For vpxd-extension, use "VPXD-EXT" as OU

After changing these values the process ran through to completion.

My last headache was cause by my reading "stopping services" and "starting services" to mean that the services had been restarted. In fact it was necessary to restart all services (or the vcenter instance in my case) before all of the new certs took effect.

I hope this helps someone in the future avoid some of the pain I've experienced for the past two days.

X-IO Technology ISE Bad Password Immediately After Reset

We installed a rack of X-IO ISE 200, and 800 series SAN shelves for POC testing purposes this week. A random password was generated to replace the default and stored in our password safe as is our procedure. Unfortunately immediately after setting this password we could no longer log in with the new one, nor the default! A bit of hammering led me to discover that this was caused by the random password containing a backslash character. As soon as I removed the backslash and attempted to log in the password worked again. Apparently the set password routine stripped the backslash (probably sanitizing input) and set the password, while the login routine treated it as a valid password character. Could that mean that the authentication fields are not sanitized at all? I hope little Bobby Tables doesn't try to log in...