Migrating to Exchange Online Office 365 using IMAP Connector - Field notes

I’ve been working on migrating customers on-premise e-mail solution to Office 365, so they could benefit from all the goodness that Office 365 offers, we encountered some issues that we couldn’t find in the official documentation. By reading the migration documentation – IMAP Migration Documentation – we thought that we planned every black scenario that could happen, but Murphy’s law happened and we faced some dreadful issues.

In this blog post, I will write about what I encountered during an IMAP migration of a Zimbra on-premise e-mail solution and what you guys should consider if you ever do an IMAP migration of a non-documented e-mail solution.

The scenario.

The customer has an on-premise Zimbra e-mail server which serves a couple of hundred users day by day. The e-mail database contains a couple of hundred gigabytes of data, and it’s used day and night for various purposes. The customer evaluated multiple e-mail service providers and ended up choosing Office 365 because of the benefits that it can add to the organization. The e-mail server has to be migrated to Office 365 as swiftly as possible with little to no downtime. Another requirement would be that users should be able to login to their new e-mail using their domain credentials.

Planning phase

Every Office 365 migration is unique. If you’ve ever done a migration, you have noticed that it’s never the same. You have a couple of scripts that do the job perfectly, but you need to modify them every damn time to take into account different problems that might occur.

So planning phase seems simple enough. We check the on-premise e-mail server, we take note of every potential problem that might occur while migrating (downtime, DNS cutoff, ISP problems, server crashes, “special needs” mailboxes, difficult mailbox owners, people that have to be kept happy at all times a.k.a priority mailboxes, meeting rooms, etc.). The second part of the ordeal is to see if the distribution groups can be exported in a suitable format. If that’s possible, then we go to the next step. Otherwise, we have a problem. The problem is that if they have lots of distribution groups that contain hundreds of members each then you can imagine what kind of problem we have on our hands, and that means manual work. From my perspective, manual work is for robots.

So we asked the question, can Zimbra export these distribution groups in a CSV format that are readable enough so we can easily create them in Office 365? The answer was that it was possible, so without further questions, we went ahead with the planning phase. Big mistake, you’ll find out why later.

Next steps in the planning phase were to discuss with the client how this migration would be handled and what we needed to do in order to achieve the lowest downtime possible:

1. Office 365 tenant created and configured (DNS Records, SPF, DKIM, DMARC, the works)
2. Prepare ADFS cluster for hybrid identity.
3. Assign O365 licenses on synchronized users.
4. Create Distribution groups and assign users to them.
5. Create Meeting Rooms.
6. Send notification to users that e-mail migration is happening.
7. Set DNS TTL to 5 minutes
8. Cutoff on-premise mail server and change MX to new O365 one.
9. Set DNS TTL back to 1H.
10. Batch-reset user passwords on the on-premise e-mail server.
11. Start IMAP migration.
12. Profit.

Sounds good? Notice any problems with the logic? Check the next part.

The problems.

Where should I start? Most of you will say, start from the beginning!

Let’s start with the easy stuff.

We created and configured the Office 365 tenant, we configured the ADFS farm with AAD Connect tool Download link, we assigned the licenses to the users, and everything was fine and dandy until I got to the distribution groups.

Earlier I mentioned that we didn’t triple check the customer’s affirmation that Zimbra could export the distribution groups in a parseable format. Well, we found out the hard way that it was parseable but only for Zimbra solutions and not for what we needed in order to create them in Office 365. If you wanted a proper format that’s parseable in PowerShell then you had to cough up a large sum of money for a converter. So what we ended up doing was the dreaded manual work to modify the .csv file in a format that could be easily used in PowerShell. That took about 10 hours of manual, tedious, heartbreaking, mind-numbing work. I tried CovertFrom-String which couldn’t be used because there was no pattern to use it, I tried regEX, I tried everything so I wouldn’t need to create the tables I needed. In the end, I gave up because there were deadlines and no time for experimentation.

So I created two .csv files, one with the Distribution Groups and one with the members of the Distribution Groups.

Here’s the code I used for this job. It’s crude, non-elegant but it works ?

$credential = Get-Credential

$exchangeSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential $credential -Authentication "Basic" -AllowRedirection
Import-PSSession $exchangeSession -DisableNameChecking

$obj = Import-Csv distri.csv
$obj.count

$obj.distributionlist | ForEach-Object { New-DistributionGroup -Name "$_" -PrimarySmtpAddress $_ }

$users = Import-Csv .\usersindl.csv


foreach ($user in $users)
{
  try
  {
    $ErrorActionPreference = "Stop"
    Add-DistributionGroupMember -Identity $user.DistributionList -Member $user.Members -Verbose

  }
  catch
  {
    "Error was $_"
    Add-Content -Value $_ -Path F:\output.log
  
  }
}

Fast forward in time, we get to the part where we had to perform the magnificent cutoff. We planned ahead in time that this operation should be done in a weekend. What could go wrong? We have 72 hours to achieve this job; it’s not like we have to shove terabytes of data into Office 365. Context -We did an assessment. Server I/O, upload Speed and with some math, we figured that we could finish the entire migration in 48 hours flat. But taking into account a lot of issues that could happen and don’t forget Murphy’s Law we agreed with the customer that the migration should be a hundred percent complete after two weeks from the start of the migration.

How did you get from 48 hours to two weeks? Well, while it was possible to do it in 48 hours, we had to take into account server failure, ISP failure, power outage, Office 365 outages, cleaning lady tripping over wires and the regular acts of God.

In preparation for the migration, we raised a ticket to Office 365 support telling them to not throttle or block us in any way. We get confirmation, and we proceed with the cutoff. We also reset the user’s passwords so we can perform the IMAP migration and we had a CSV standing by to instruct the Exchange Online IMAP Connector what to do.

We start the IMAP migration, and we check the on-premise server to see if it’s getting any connections and it started transferring data to Office 365. Funnily enough, it worked fine for the first few hours, and then we refresh the Exchange Online page, and that’s where the problems started. First of all, Zimbra is a freak of nature of an e-mail solution, and the IMAP connector for migrating data into Office 365 is closing in fast. The whole 150 batch migration job failed because all the mailboxes contained trash, contacts, distribution lists, calendar, junk and notes folders which couldn’t be migrated. So what’s the point in migrating what we can, when we have the possibility to stop the job and let you know after two hours? We get frustrated, fix the batches and restart.

Next in line in the form of errors were the bad item and large item limits were reached, so Exchange killed the jobs again, which in the end made us turn to old wonderful PowerShell. We didn’t start with PowerShell in the beginning because the GUI was very easy to use and why bother?

So we went with ol’faithful PowerShell and we restarted the batch migration using the code below:

$credential = Get-Credential

$exchangeSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential $credential -Authentication "Basic" -AllowRedirection

Import-PSSession $exchangeSession -DisableNameChecking

New-MigrationBatch -Name Batch_150_1 `
                   -SourceEndpoint "IMAP Non SSL " `
                   -CSVData ([System.IO.File]::ReadAllBytes("d:\Dropbox (Personal)\CSV\mailboxes.csv")) `
                   -AutoStart `
                   -BadItemLimit unlimited `
                   -LargeItemLimit unlimited `
                   -ExcludeFolders "Contacts","Distribution Lists","Trash","Calendar","Junk","Notes" `
                   -NotificationEmails "[email protected]"`
                   -Verbose

What happened next you ask? IT WORKED! Until.. we hit a hard limit. There’s throttle limit in Office 365 which cannot be turned off, and that limit says that you can upload 500MB of data per 24 hours per mailbox and no more. Why didn’t we know this before we started planning? Because it was easier to import the high profile and large mailboxes using the PST import migration tool. I only dealt in extremes (large and small mailboxes) and it’s not documented anywhere.

What did we end up doing? We kept the original server up and running and we granted access to the users that required access to their old mailbox. The migration finished in ten days. We had everything prepared, but Office 365 throttled us to death.

In the end, the migration finished, and we learned some things. You may say that it was a rookie mistake to do the cutoff before the sync, and I agree, but if we’ve known about that 500MB/Day/Mailbox limit, then the migration would have happened after the initial sync. If I would do the migration again, the only thing that I would change is the cutoff part. Do the initial sync, do the cutoff, sync leftovers and profit ?

Hope you enjoyed my little post here, and as always have a good one!