Severe Outlook Performance Issues Connecting to Exchange Online in RDS 2012 R2

O365AdminM08_960

If you are experiencing severe issues with Outlook performance in RDS 2012 R2, this article is if for you. This article represents several weeks of work and stress. After migrating user mailboxes to Office 365 were encountered an issue that appeared so impossible that we were even considering moving the entire organization back to on-prem. It took Microsoft support engineers over 2 weeks, bless their hearts, to finally provide a fix that actually worked. It took another week to test it and several days to implement. First I will walk you through a little history.

The environment:

  • Outlook 2013
  • RDS 2012 R2 deployment
  • Folder redirection and UPD for Appdata
  • On-prem Exchange 2010
  • All hosted in a colocation site
  • 230+ network users
  • Hybrid deployment with Exchange Online in Office 365

We deployed Office 365 in starting back in 2013 and took some time to get to know the interface and features while continuing to use our on-prem Exchange 2010 server. In the summer we decided to move forward with a hybrid deployment to migrate user mailboxes at our own pace  to avoid the pressure and rush of a cutover migration.

Over two months of slowly moving mailboxes and testing,  we found 365 to be a touch slower to load the message in the view pane but it was barely noticeable. So we decide to go for it. We migrated all the rest of the user mailboxes to Exchange Online. Users we so proud that they could tell their friends that they were in the cloud!

Users continued to use Outlook 2013 in RDS to connect to Exchange Online. For two weeks we did not have a single complaint! Users barely noticed except that they had to change the settings on their phones to get email. We recommended users download OWA for Android and iPhone. The app was well received because it look and feels just like OWA in 365. Then all of a sudden on Friday August 22nd after lunchtime, we got slammed with support tickets. The CEO said “my computer is freezing up” and sure enough Outlook was hanging bad.

While not every user in our office had issues right away, the impact was severe enough to be highest priority. Office 365 was unaffected. Users did not have any problems accessing email through OWA but they were not used to the interface and several refused to use 365.

Before involving Microsoft support engineers I tried all the basic steps from this Microsoft KB article on basic Outlook troubleshooting and several more. Microsoft engineers suggested fixes like: using the Microsoft Office Configuration Analyzer Tool 1.2 (OffCAT) to find issues. I disabled the hybrid configuration for a while to test if that was the cause, it wasn’t. During this process we upgraded Outlook to the latest version with patches that added some features. After a week I had spoken with so many support engineers from every team that I knew more than most of them did, including: the Outlook team, RDS team, mailflow team, Exchange Online team , and hybrid team – and opened I multiple tickets too!

Anytime I got pitched to a new team, the green tier 1 engineer would fumble around so much I was telling them what to do. Many of the engineers had no idea how to approach this issue because they were unfamiliar with RDS technology,  the layer of complexity RDS presented or could not get around the inability to use normal troubleshooting tools in that environment. For example the normal Microsoft “Fix it” tools were throwing errors.

Several engineers suggested using cached mode although it is not advised for RDS and when we got really desperate, we instructed the users with issues to set their mail profiles to cached mode. This is a great Microsoft article on when to use cached mode. Finally I reached a tier 3 engineer in the Exchange Online department, he determined that we needed to go back to work with an engineer in the Outlook team. The quest came full circle.

In the end, when working with the Outlook team we ran Procdump (tricky in RDS because limited user rights settings) to gather the needed data from a problem account. After a week of analyzing the results, the engineers had a potential solution.

Outline of the issue experienced:

Outlook says Not Responding in the application title bar, app whites out and is inaccessible

Outlook Not Responding Error

User sees message in lower right saying: Outlook is trying to retrieve data from the Microsoft Exchange server outlook.office365.com

Outlook trying to retrieve data error

Outlook does not load profile when opening or takes 5-10 minutes to load

Outlook mailbox open error

If you click too much while it is loading you get an apphang and system freezes, giving various errors

Outlook trying to retrieve data network error

Not all users had issues, and the issue was not 100% repeatable but for problem accounts usually I could get the error by just opening Outlook and clicking around too fast. It was a come-and-go issue, inconsistent and variable but impacting a significant amount of users.

The resolution 

Disable MAPI/HTTP by setting the following key on the RDS session host server. Open regedit and navigate to this location:

HKEY_CURRENT_USER\Software\Microsoft\Exchange
Create a DWORD entry with the name MapiHttpDisabled
Value: 1

DWORD

After creating the DWORD, open the entry and enter the value 1

Result

After Making the above changes please restart Outlook and go to the connection status Windows by holding the Crtl Key down and right clicking the Outlook icon in the System Tray.

In the window that Pops-up > scroll to the Right and under connections ensure it is displaying Shows RPC/HTTP.

Ensure that there is only a single account configured in Outlook and check if the issue persists. You may want to restart the users RDS session to refresh everything before testing.

For most users this fix worked great immediately, but for a single user where we had to delete and create his UPD (user account appdata) so it would forget his corrupted profile and build a new one for him. After that it worked great!

All the while through this issue the support engineers at Microsoft were acting baffled, like this was the first time they had ever seen this issue, and for many that may have been true. But later on when communicating with a really great engineer on the mail flow team for a different issue, he said, “oh that, that is a known issue.”