Migrating Mailbox to Cloud Fails in Hybrid Exchange Deployment

If you attempt to migrate an on-prem mailbox to Exchange Online using the Office 365 portal and it fails, there are several steps to take to correct the issue. This article assumes you have DirSync installed and are using it to sync your passwords.

Start by checking that the user has an Exchange license in Office 365 by using the portal. Lack of an Exchange Online license may cause migration failures.

Migrating 5

To check, go to the Admin page in the Office 365 portal

Migrating 1

Find and select the user

Migrating 2

 

On the user profile page, go to licences and select the location and both the Exchange Online and SharePoint Online licenses, save

Migrating 3

This may solve the problem. Wait a minute or so and attempt to migrate the mailbox again. If it doesn’t work, move on to the next steps below which are more complicated.

Migrate mailbox from on-prem Exchange to Office 365 with powershell 

First step is to download and install Windows Azure Active Directory Module for Windows PowerShell. Once installed, start the PowerShell module on a domain joined machine.

ad azure ps

 

To connect to a session, cut and paste these following PowerShell commands all at once, hit enter when needed to execute. The first two prompts require Office 365/Azure global admin credentials, in full email or “domain\username” form. The last prompt requires local admin credentials in full email or “domain\username” form.

Import-Module MSOnline
Connect-MsolService
$LiveCred = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection
Import-PSSession $Session
$OnPremAdmin=Get-Credential

 

Severe Outlook Performance Issues Connecting to Exchange Online in RDS 2012 R2

O365AdminM08_960

If you are experiencing severe issues with Outlook performance in RDS 2012 R2, this article is if for you. This article represents several weeks of work and stress. After migrating user mailboxes to Office 365 were encountered an issue that appeared so impossible that we were even considering moving the entire organization back to on-prem. It took Microsoft support engineers over 2 weeks, bless their hearts, to finally provide a fix that actually worked. It took another week to test it and several days to implement. First I will walk you through a little history.

The environment:

  • Outlook 2013
  • RDS 2012 R2 deployment
  • Folder redirection and UPD for Appdata
  • On-prem Exchange 2010
  • All hosted in a colocation site
  • 230+ network users
  • Hybrid deployment with Exchange Online in Office 365

We deployed Office 365 in starting back in 2013 and took some time to get to know the interface and features while continuing to use our on-prem Exchange 2010 server. In the summer we decided to move forward with a hybrid deployment to migrate user mailboxes at our own pace  to avoid the pressure and rush of a cutover migration.

Over two months of slowly moving mailboxes and testing,  we found 365 to be a touch slower to load the message in the view pane but it was barely noticeable. So we decide to go for it. We migrated all the rest of the user mailboxes to Exchange Online. Users we so proud that they could tell their friends that they were in the cloud!

Users continued to use Outlook 2013 in RDS to connect to Exchange Online. For two weeks we did not have a single complaint! Users barely noticed except that they had to change the settings on their phones to get email. We recommended users download OWA for Android and iPhone. The app was well received because it look and feels just like OWA in 365. Then all of a sudden on Friday August 22nd after lunchtime, we got slammed with support tickets. The CEO said “my computer is freezing up” and sure enough Outlook was hanging bad.

While not every user in our office had issues right away, the impact was severe enough to be highest priority. Office 365 was unaffected. Users did not have any problems accessing email through OWA but they were not used to the interface and several refused to use 365.

Before involving Microsoft support engineers I tried all the basic steps from this Microsoft KB article on basic Outlook troubleshooting and several more. Microsoft engineers suggested fixes like: using the Microsoft Office Configuration Analyzer Tool 1.2 (OffCAT) to find issues. I disabled the hybrid configuration for a while to test if that was the cause, it wasn’t. During this process we upgraded Outlook to the latest version with patches that added some features. After a week I had spoken with so many support engineers from every team that I knew more than most of them did, including: the Outlook team, RDS team, mailflow team, Exchange Online team , and hybrid team – and opened I multiple tickets too!

Anytime I got pitched to a new team, the green tier 1 engineer would fumble around so much I was telling them what to do. Many of the engineers had no idea how to approach this issue because they were unfamiliar with RDS technology,  the layer of complexity RDS presented or could not get around the inability to use normal troubleshooting tools in that environment. For example the normal Microsoft “Fix it” tools were throwing errors.

Several engineers suggested using cached mode although it is not advised for RDS and when we got really desperate, we instructed the users with issues to set their mail profiles to cached mode. This is a great Microsoft article on when to use cached mode. Finally I reached a tier 3 engineer in the Exchange Online department, he determined that we needed to go back to work with an engineer in the Outlook team. The quest came full circle.

In the end, when working with the Outlook team we ran Procdump (tricky in RDS because limited user rights settings) to gather the needed data from a problem account. After a week of analyzing the results, the engineers had a potential solution.

Outline of the issue experienced:

Outlook says Not Responding in the application title bar, app whites out and is inaccessible

Outlook Not Responding Error

User sees message in lower right saying: Outlook is trying to retrieve data from the Microsoft Exchange server outlook.office365.com

Outlook trying to retrieve data error

Outlook does not load profile when opening or takes 5-10 minutes to load

Outlook mailbox open error

If you click too much while it is loading you get an apphang and system freezes, giving various errors

Outlook trying to retrieve data network error

Not all users had issues, and the issue was not 100% repeatable but for problem accounts usually I could get the error by just opening Outlook and clicking around too fast. It was a come-and-go issue, inconsistent and variable but impacting a significant amount of users.

The resolution 

Disable MAPI/HTTP by setting the following key on the RDS session host server. Open regedit and navigate to this location:

HKEY_CURRENT_USER\Software\Microsoft\Exchange
Create a DWORD entry with the name MapiHttpDisabled
Value: 1

DWORD

After creating the DWORD, open the entry and enter the value 1

Result

After Making the above changes please restart Outlook and go to the connection status Windows by holding the Crtl Key down and right clicking the Outlook icon in the System Tray.

In the window that Pops-up > scroll to the Right and under connections ensure it is displaying Shows RPC/HTTP.

Ensure that there is only a single account configured in Outlook and check if the issue persists. You may want to restart the users RDS session to refresh everything before testing.

For most users this fix worked great immediately, but for a single user where we had to delete and create his UPD (user account appdata) so it would forget his corrupted profile and build a new one for him. After that it worked great!

All the while through this issue the support engineers at Microsoft were acting baffled, like this was the first time they had ever seen this issue, and for many that may have been true. But later on when communicating with a really great engineer on the mail flow team for a different issue, he said, “oh that, that is a known issue.”

 

How to Prepare AD for Exchange 2013 Server Install in Hybrid Deployment

If you have a current Exchange 2010/Office 365 hybrid deployment with  you may want to consider adding at least one Exchange 2013 server. Microsoft will provide you with one for free, but disallows hosting mailboxes on the Exchange 2013 server with the complimentary license. Read about how to get your free Exchange 2013 license here.

office 365

Preparing Active Directory for a hybrid configuration is a slightly different than a non-hybrid configuration. Follow these comprehensive steps for a quick solution to prepare your Active Directory for installing Exchange 2013 Server on-prem when you have a pre-existing hybrid deployment in your organization.

First objective is to get an XML file with your hybrid configuration information. To obtain the configuration files, open Azure Active Directory Module for Windows PowerShell.

  • If you need the module, use the link above, download the 32-bit or 64-bit version and install it on a domain joined computer or server.

Log into a domain controller (DC) and create a folder on the C drive named “a” just to keep it simple. Open elevated session in Azure Active Directory Module for Windows PowerShell (Run as Administrator).

run as

Connect to Azure AD using commands below, cut and paste all at once and when the windows pops up enter global admin account credentials for your 365 tenant.

$LiveCred = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection
Import-PSSession $Session

powershell creds

ps 2

Once connected you need to obtain the XML config file, enter this command:

Get-OrganizationConfig | Export-Clixml -Path c:\a\MyTenantOrganizationConfig.XML

If it doesn’t work you are either not connected to Azure AD or you didn’t create a folder in your C drive called “a” for the file to be placed in. Once you have it figured out continue.

  • Warning you will not be able to install any new Exchange 2007 Servers in your environment after following the next steps.

Download and unzip the Exchange 2013 installation files. I placed them in a folder called “ex 2013 install” on the C drive of the DC but you can call the folder whatever you want.

After verifying that the XML file is in the “a” folder on your C drive, open an elevated session of PowerShell and run this command to prepare your AD.

.\setup /PrepareAD /IAcceptExchangeServerLicenseTerms /TenantOrganizationConfig c:\a\MyTenantOrganizationConfig.XML

ad

After the preparation is completed on the DC, login to the server you intend to install Exchange Server 2013 on and continue your installation.

ex2013

Read about how to get your free Exchange 2013 license here.