Exchange hybrid – mailboxes missing on-premise

While hybrid exchange environments are awesome for stretching your on premise exchange topology to Office 365, they do introduce a bunch of complexity – primarily around user creation, licensing, and mail flow.

I recently had an issue at a client where they had email bounce-backs from an on premise service destined for a few Exchange Online mailboxes. For some reason, these few mailboxes didn’t appear in the on-premise exchange environment (as remote Office 365 mailboxes), so exchange was unable to route the emails destined for those particular mailboxes.

In general, you should be creating your mailboxes on premise (Enable-RemoteMailbox), then synchronising via AADConnect – that way the on premise environment knows about the mailbox and it can be managed properly. This client was actually doing this, but obviously the process broke somewhere along the way for a few mailboxes.

There’s a bunch of different options on Google about how to get the mailbox to show up on premise – with a lot of them recommending to remove the mailbox and start again (er… how about no!).

I came across this Microsoft article on a very similar issue, but for Shared Mailboxes created purely in Exchange Online. Looking at the process, it looked like a modified version may work for user mailboxes – and it does. Below is a quick and dirty powershell script that can be used to fix a single mailbox:

#Specify who we're working with
$UPN = "end.user@domain.com"
#Local exchange server
$ExServer = "Server1.local"
#365 Domain - for remote routing address
$RoutingDomain = "mydomain.mail.onmicrosoft.com"

#Connect to 365 Exchange - only import select cmdlets so they don't conflict with the Exchange On Premise session
$RemoteSession = New-PSSession -ConfigurationName Microsoft.Exchange `
      -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $(Get-Credential) `
      -Authentication Basic -AllowRedirection
Import-PSSession $RemoteSession -CommandName Get-Mailbox

#Connect to local exchange - only import select cmdlets so they don't conflict with the Exchange Online session
$LocalSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "http://$ExServer/PowerShell/" `
      -Authentication Kerberos -Credential $(Get-Credential)
Import-PSSession $LocalSession -CommandName Enable-RemoteMailbox, Set-RemoteMailbox

#Get the Alias and ExchangeGuid from 365
$Mailbox = Get-Mailbox $UPN
$Alias = $Mailbox.Alias
$ExchangeGUID = $Mailbox.ExchangeGuid

#Create a remote mailbox
Enable-RemoteMailbox $UPN -Alias $Alias -RemoteRoutingAddress "$Alias@$RoutingDomain"
#Set the Remote Mailbox GUID to match the 365 mailbox GUID
Set-RemoteMailbox $Alias -ExchangeGuid $ExchangeGUID

#Remove sessions
Get-PSSession | Remove-PSSession

 

Migration of public folders to exchange 2013/2016

I’ve done a few of these…. but most corporates (at least that I’ve dealt with) use public folders quite lightly – if at all…. so the migrations have been quite simple.

 

Recently, I was tasked with moving a smaller business (through a partner) from 2007 to 2013 then 2016.

The mailbox move from 2007 to 2013 went flawlessly.

Then we came to their public folders…. approx. 400GB – from which they apparently run a lot of their business.

Ran through the (painful) process of removing trailing spaces, backslashes, dead permissions etc… not hard – just slow, manual and annoying.

There is an article here that talks about the hassle of migrating PF’s – https://thoughtsofanidlemind.com/2013/12/13/migration-modern-public-folders/

On the first migration attempt, the extent of these corrupt items and oversize items was discovered (3000 corrupt items and hundred’s of items that were oversize) – then discussed with the business.

So here we have the first fucking boomingly huge issue with public folder migration…. there are no powershell commandlets to help you get the size of items (you can get the size of folders, but that’s not helpful) that will be considered oversize… so you cannot identify these items prior to migration. To add to that, even if you could identify them, there is no nice way to say “export these items to PST, then delete” or as part of the migration batch “migrate all large items”

The next issue here is that through the GUI, you can see a list of skipped items and why they were skipped (corruption or oversize) – there doesn’t appear to be way to get this information via powershell so you nicely export it and give it to the customer (or sort it yourself)

The business stated that corrupt PF’s weren’t vital, but the large items were needed.

Even after lifting the size limit to 500MB, there were still lots of items that were too large.

I tried to accommodate these large items and found the exchange migration mailbox (a default database which I leave in the default location) – which should only ever be used in transit, proceeded to grow, fill up the disk that logs were on and cause a dirty shutdown and corruption…  so I haven’t my lesson there…. if a client is using PF’s as a file store for items of 500mb over – refuse to migrate until these items are removed…. (unfortunately you need run a “dummy migration” then look at the skipped items list to identify these items!)

Anyway – long story short – the moral of this, very annoyed, story

Public folder migration to Exchange 2013/2016 sucks. It has clearly been put in as an after-thought to appease some organisations – and is only suitable for light users of PF’s

If a customer is a heavy public folder user, do not change the default “large item” size to accommodate them. Refuse to migrate them and notify them the items will be lost.

 

Microsoft Exchange Federation Certificates – Keep an eye on the expiry!

I recently had a client experience an issue with their hybrid exchange setup (365/On Premise) – users were suddenly unable to retrieve free/busy and calendar information between the two environments. As it turns out, the certificate used to secure communications to the Microsoft Federation Gateway (MFG) had expired.

Federation certificates within exchange are generally created as part of the federation creation wizard (or the 365 Hybrid Configuration Wizard) – so in most cases, people don’t realise they’ve been created. If you’re not actively monitoring certificate expiry dates on your servers (which you should be!), you may get into the situation where this certificate expires – which results in the federation no longer working.

Why is it important to renew it before it expires? Because if you don’t, you need to remove and re-create the federation – a significantly larger task than the federation certificate renewal process. The reason for needing to re-create the trust is due to the fact that the federation certificate is used to authenticate any changes to the federation – so once it expires you can’t make any changes and have to start from scratch. Lets take a look at the steps involved in both:

Renewing before expiry:

  1. Create a new self-signed federation certificate
  2. Set the new certificate as the ‘Next’ certificate in the federation trust
  3. Wait for AD replication
  4. Test the certificate and trust (Test-FederationTrustCertificate, Test-FederationTrust)
  5. Roll-over the ‘Current’ certificate to the ‘Next’ certificate
  6. Refresh the federation metadata

Renewing after expiry:

  1. Document the existing trust settings (federated domains, federation settings)
  2. Force remove each federated domain from the federation
  3. Remove the federation trust
  4. Wait for AD replication
  5. Create a new self-signed federation certificate
  6. Create a new federation trust
  7. Update the trust organisation information
  8. Configure the required settings in the trust (as per the documentation you created in step 1)
  9. Wait for AD replication
  10. Test the certificate and trust (Test-FederationTrustCertificate, Test-FederationTrust) – it can take 12-48 hours before the trust reports as being no longer expired!
  11. Add each of the federated domains back into the trust (this will involve generating domain ‘Proof’ entries and adding them to your external DNS, then waiting for DNS propagation)

So in short, don’t let your federation certificates expire!

Exchange 2013/216 receive connector selection process

On a recent gig, the local tech wanted some more detail on the receive connector selection process… I explained the whole “most defined” IP set wins (i.e. 192.168.0.23 will be selected over 192.168.0.0/24), but was unable to find any TechNet articles to back that up.

Fortunately, this guys nails it, with a good explanation and examples…

http://markgossa.blogspot.com.au/2016/01/receive-connector-selection-exchange-2013-2016.html

good work Mark Gossa.

 

KB3124557 sets exchange 2013/2016 services to disabled

A patch for exchange 2013 and 2016 sets all exchange services (and IISAdmin and W3SVC) to disabled after installing

https://technet.microsoft.com/en-us/library/security/ms16-010.aspx

Its not a disaster, just simply set all the services back to automatic, start them, and away you go….

Still, the joke conspiracy from a mate of mine a while back of “The marketing team are forcing the exchange team to fuck up updates, as a way of pushing people to exchange online”, based on the amount of botched updates, is becoming more and more plausible.

Handy list of antivirus exclusions for MS products

http://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx

I am still very much of the opinion that file level anti-virus does not belong on servers such as exchange, SQL, SCCM etc… however, for places that flat-out refuse to apply logic – and want anti-virus on everything, this is a very good starting point to apply the mountains of exclusions required.

Direct Access 2012 and Outlook RPC/HTTP

I have a client for whom i recently implemented 2012 Direct Access…. there was a number of teething problems in the implementation – but once its working, DA is actually pretty good.

If i could ask the product team for anything in DA 2012 R2, it would be

  • re-introduce wildcard certificate support (it was in UAG DA for 2008 R2, not sure why it got left out of 2012)
  • Make the status reporting dashboard check for local firewall status (that its on and that the rules have been correctly added) as part of its health check
  • Release Win7 SP2 and make it support “simple” DA auth (i know this isn’t going to happen… but it doesnt mean it wouldn’t be nice!)

Anyhoo….. at this particular client, they had decided to do one of my most hated things – use the same url internally and externally for the configuration of web services in exchange. I avoid this configuration where-ever possible…. although i understand at small sites where only one server is present – it may seem like the only option – solutions such as TMG (or another reverse publsiher), SAN certificates or creating additional exchange web services on different ports are all valid ways of getting around that config.

As to why it had been done at this client who is substantially larger…. well, the consultancy that did it…. i have very different views to them on good practices for exchange design…. and the client has a DAG without a load balanced CAS – so there is no true fail-over anyway…. grrr…. but i digress.

Due to the configuration of the same url’s internally and externally for web services, i added an entry into the DA NRPT so it would always resolve to the internal name…. issue is, that users were still getting prompted for auth, because the outlook client detected a slow connection and used RPC/HTTP, which is configured for basic auth (another config item i diagree with).

What i had not know before (because i hadnt needed to) was that you can disable RPC/HTTP on a per-mailbox basis…. using the command

Set-CASMailbox –Identity <alias> –MAPIBlockOutlookRpcHttp $True

While the better solution is to fix the exchange design (different URL’s and NTLM for RPC/HTTP) – its a handy work-around.

Exchange 2013 Gotcha’s

Found this article today

http://theessentialexchange.com/blogs/michael/archive/2013/01/06/exchange-server-2013-gotchas.aspx

it’s nigh on 3 months old – but it lists some of the issues with Exchange 2013 RTM.

Prior to finding this article – I hadn’t been able to find any list of issues with the release, but my own list of issues was growing.

Exchange 2013 release seems to have been rushed a little, but lets hope that these issues are addressed by CU1 (which is due anytime now) – and I’m really hoping the EMC makes it back in by SP1 (but I doubt it!)

Exchange 2010 recovery scenario – OS disk dead

Exchange recoveries don’t come up very often these days (for me at least), with exchange 2010 and win 2008R2 being so solid… and virtualisation… the loss of server hardware or corruption of the OS is just becoming less and less common.

Anyhoo – a client had a SAN disk die – and for reasons which I didn’t delve too deeply into, apparently the RAID5 didn’t work and they lost one vdisk…. (yes, it sounds suss to me too…. this was a EVA4400… so not a cheap SAN)  this vdisk happened to have the exchange OS partition on it…. but the database and log disks were fine….

So, we built up a server with the same name as the old one, gave it the same IP and joined it the domain…. the connected the logs and database drives to the newly built server.

Using ADSIEdit, I then looked up the path to the database, as the client could not remember which drive had which letter – if your in this situation, fire up ADSIEdit and connect to the configuration partition, then navigate to CN=DatabaseName,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ORGNAME,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=blah,DC=au

have a look at the properties msExchEDBFile and msExchESEParamLogFilePath…. this will help you allocate the drive letters correctly should you run into a site that doesn’t know this or have it documented.

Next issue – versioning…. the client wasn’t sure which Service Pack version exchange 2010 was at, but though it was SP2.

I grabbed the SP2 media and after installing the pre-reqs, ran setup /mode:RecoverServer

First error came up for the CAS and HT roles…. must install a version of 14.1.218.5 or later….. that number relates to Exchange 2010 SP1 RTM…. funny, SP2 is later than SP1… but oh well…. SP1 it is….

So I grabbed the SP1 media and re-ran the setup command…. and proceeded to get

[ERROR] The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell

because there is not another exchange server within this environment, I cant simply remove the edge transport from the GUI… so I went searching in ADSIEdit to

CN=HTServer,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ORGNAME,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=blah,DC=au

Go to the properties of the HTserver container and the properties of the property “msExchEdgeSyncCredential” – remove the 3 entries.

Try the recovery again, and get

     A Setup failure previously occurred while installing the HubTransport role.

Either run Setup again for just this role, or remove the role using Control Panel.

Awesome…. try to uninstall via control panel, no go, try to uninstall via command line, no go…. so, the solution:

  • Open up regedit
  • Navigate to HKLMSoftwareMicrosoftExchangeServerV14
  • Delete the “Action” and “watermark” keys

Now re-run the install – all is good…. yay!

Next up, re-allocate the certificates on the server, re-associate the edge transport – and test (and then upgrade to SP2 RU5v2 as of time of writing!)

All in all, the recovery procedure is still good, but complicated a bit by the edge transport being present (im not a fan of edge transports…. use a HT with forefront for exchange with port 25 published via TMG instead….. or if you prefer, an ironport)