Lotus Notes to SharePoint Blog

Blog about Dell's Notes Migrator to SharePoint tool and other things related to Lotus Notes migration projects

Migrating to the SharePoint Online platform (BPOS)

[Update: With the launch of Office 365, this article is somewhat out of date.  Notes Migrator for SharePoint now supports migrating directly to both the Standard and Dedicated flavors of Office 365 via standard Sharepoint 2010 web services.  For the Dedicated offering, most customers will still elect to use Option 1 or (more likely) the Hybrid approach listed below.  See https://notes2sharepoint.org/category/sharepoint-online/ for more details.]

If you are talking to Microsoft about getting off of Lotus Notes, I have no doubt that you are hearing about, and probably seriously considering, Microsoft’s Business Productivity Online Suite (BPOS), also referred to as Microsoft Online (MSO).  There is a Dedicated version (BPOS-D), which give large enterprises the opportunity to have private hosted servers dedicated to just them, and a Standard version (BPOS-S) for everyone else.  SharePoint Online is, of course, part of these suites.

There is a ton to say about the Dedicated and Standard offerings, most of which I will save for another day, but here I want to focus on migrating the content of your legacy Notes applications to the cloud.  I have not blogged about it much, but I have been deeply immersed in this issue for over a year now – working with various Microsoft teams, helping customers think about and then conduct such migrations, and (of course) working to make Notes Migrator for SharePoint an ever stronger tool for this job.

As you will see below, there are now three good options to consider when migrating to SharePoint Online Dedicated and really only one for Standard.  For Dedicated, there are many trade-offs and factors to consider and some customers are looking at a hybrid approach as discussed at the end of the article.  SharePoint 2010 is definitely changing the game here (both because of the new capabilities of the platform and the new migration options) and many of our customers are waiting for the 2010 releases of SharePoint Online to begin serious migrations.

As you read this, note that much of this discussion is not really specific to BPOS.  SharePoint Online is in many ways really just a very secure environment with very high expectations of reliability and uptime, not all that different from what we see our large enterprises customers trying to deploy internally.  These options are all worth considering for any large enterprise migration.

Option 1: Bulk migration via a staging environment (Dedicated only)

The first option is to use an on-premises staging environment.  This could be as simple as a single server on a VM or a beefy multi-server farm, preferably on the same WAN as your old Notes/Domino environment.  You install the Notes Migrator for SharePoint Import Service here and do all migration work inside your firewall.  If needed, you would also do the bulk of your site development and customization work in this environment, as well as end-user acceptance testing.  When you are ready, you simply detach your site’s content database from your on-premises SharePoint farm, FTP or FedEx it (depending on the size) to the Microsoft data center, and Microsoft will add it into your hosted SharePoint farm.  Even when doing pure on-premises migrations, many of our customers prefer to use a staging environment as a best practice anyway, so this is a strong option to consider.

Advantages

  • Performance and bandwidth – If you have lots of data to migrate (and some of our customers have terabytes!) the time it would take to send it all over the wire becomes prohibitive.  It is much faster to migrate locally and then ship the content databases.
  • Easier to manage and configure things – Once your data goes live, you will no longer to be able to access Central Administration directly; if you need something changed, submit a change request.  While things are still in staging, you have the ability tweak the CA settings and other things that might impact your migration.
  • Test environment – Of course a staging environment can also be a test environment, which most people want anyway.  Even if you are not writing custom code, you will probably want to configure your target sites, customize lists / libraries / pages / web parts, and customize your migration jobs to match.  This often works best as an iterative process done in a staging environment rather than a live production environment.
  • Simplicity – This is the option that Microsoft usually recommends first and for good reason.  There is no need to approve and install additional software in the production environment, think about the bandwidth issues during an intense migration phase, grant administrative access to the migration team, etc.  From the production team’s perspective, all these things represent risk.  Better if the migration team does all their work in the staging environment and the deployment team does not have to think about anything other than plugging in the content databases.

Disadvantages

  • Granularity – Promoting content to production by moving content databases tends to be an all-or-nothing event.  You have to get every list in every site in the content database exactly right, validate that the users are happy with the result and that you did not miss any of the content, and then pull the plug.  If you find out a month later that one of your team rooms had a custom field that you did not know about, you will not have the opportunity to re-migrate to the same location. 
  • Timing – Related to the granularity issue above, you have to wait for the entire content database to be migrated before you can “go live” with the first list.  On top of that you have the time required to ship the content to Microsoft and wait for the next change window.  All this can add up to weeksbetween the last day a user is allowed to enter content in the old Notes application and when the user can enter new stuff in the online version of that application.
  • Surprises – SharePoint Online has some limitations and it is possible that some of the features, web parts, connectivity options, etc., that you designed into your new SharePoint site will not be allowed in production.  These same limitations would be there regardless of which migration method you used, of course, but with this option there is a risk that you would not find out about it until the day you were expecting to go live in production.  The remedy is to study the documentation carefully and do trial migrations well ahead of time.
  • Link Finalization – If you are using Quest’s innovative Link Tracking Service, you are likely to want to “finalize” all your Dynamic Links (i.e. replace them with permanent direct links) once the migration is finished.  Of course you will probably finalize as many links as you can in your staging environment before promoting your content, but in large migrations there will always be exceptions.  The documents that you migrate this month are likely to contain at least some links to the documents you plan to migrate next month.  You will not be able to finalize such links using Option 1 alone.

Option 2: Direct migration via Quest’s Import Service (Dedicated only with custom approval required)

The second option is to install the Notes Migrator for SharePoint Import Service on your hosted production SharePoint farm.  To someone who is experienced with our product, this may seem like a no-brainer.  We have a mature, elegant, client-server solution in which the components talk to each over secure, efficient web services.  So it only makes sense to install the server bits on the server, right?

Well there is one big problem with this approach: Microsoft is (understandably) very, very careful about what they allow to be installed on their servers.  They are concerned about the fact that there is third-party code running on their servers, the fact that they may need to get involved in configuring and troubleshooting this service, and the fact that a period of intense migration traffic may tax the servers beyond the normal usage that the platform was designed to support.

For these reasons, installation of tools such as the Notes Migrator for SharePoint Import Service is treated as a special “customization” requiring a special approval process.  The end-customer needs to submit the paperwork (with Quest’s help, of course) and participate in the testing and approval process.  This is a lot of extra work for all parties and we encourage customers to carefully consider whether it is worth the trouble.  It can be done – and our customers have gone through it and lived to tell the tale – but if you can tolerate the limitations of Option 1, or wait or Option 3, doing so will save you time and energy.

Advantages

  • Real-Time Migrations – You can use the Notes Migrator for SharePointclients to design a  migration job and immediately run it.  This includes not just content migration, but also provisioning of target sites, lists and libraries, custom schema, security and more.  You and all your stakeholders can see what the result looks like immediately.  Best if all, if you want to adjust something you can make a change to your migration job and run it again.
  • Incremental Migrations – In the real world, migrations are not always all-or-nothing.  Sometimes there is a coexistence period in which some users are still entering things in Notes even after others have moved over to SharePoint.  Your project management system in SharePoint may reference a Customer list that is still being maintained in Notes and synched to SharePoint on a nightly basis. These types of scenarios are very hard to accomplish with Option 1 alone.
  • Flexibility – The ability to migrate one list (or even one document) at a time can be very important.  This flexibility allows you to arrange your migration process according to your business needs and priorities, rather than the physical arrangement of which sites are in which content databases, etc.
  • Full functionality – All the awesome features of Notes Migrator for SharePoint are available to you, including the Link Tracking service and the ability to finalize your links at any time.  In addition, ad-hoc user-driven migration may be performed.

Disadvantages

  • Approval Required – As described above, the use of Notes Migrator for SharePointrequires going through Microsoft’s process for customizations.  This is the same process you would have to go through if you developed your own add-ins that need to run outside of the SharePoint 2010 “sandbox”.  As the BPOS-D teams typically run on a quarterly cycle, the timing of the approval process can become an issue.
  • SLA Modifications – Because the Notes Migrator for SharePointImport Service is architected as a stand-alone web application (rather than a SharePoint solution) it makes the approval process even harder.  Microsoft may insist on suspending or modifying your standard Service Level Agreements during the migration period.
  • Bandwidth – Don’t expect to migrate terabytes of data over the Internet to the Notes Migrator for SharePoint Import Service (or to any web service).  People with that kind of volume definitely need to look at Option 1 or a hybrid approach (see below).

Option 3: Direct migration via enhanced web services (SharePoint 2010 releases only)

SharePoint 2010 has a greatly expanded set of web services that are (finally) adequate to the job of migrating content.  (Yes, SharePoint 2007 had web services too, but the limitations were too great for most Notes applications.  For example, it was not possible to migrate a document and preserve the document metadata such as the author name and the created-by date.)  With this option, Notes Migrator for SharePoint can provision things, migrate high-fidelity content, set permissions, etc. all via Microsoft’s out-of-the-box interfaces.  You get the best of both worlds:  real-time content migrations and no additional code needed on the production environment.

IMPORTANT: At the time of this writing, neither SharePoint Online Dedicated or SharePoint Online Standard have released their SharePoint 2010 versions, so I need to stipulate that things are subject to change here.  I do not think that I am liberty to discuss Microsoft’s release schedules, but I will say this much:  If you are participating in the TAP or beta programs for either platform, we would love to work with youIf interested, please contact your Quest sales rep or reach out to me directly at steve.walch @ quest.com.

Advantages

  • Best of both worlds – This option gives you most of the advantages of Option 2 without the need to get anything approved by Microsoft.

Disadvantages

  • Not available yet – As described above, you have to wait for future releases of SharePoint Online to use this method.
  • Throughput – This solution is not as fast as the Notes Migrator for SharePoint Import Service, which was designed expressly for Notes migrations.  The Microsoft interfaces were designed for a variety of purposes (such as building high-end client interfaces) and are necessarily more “chatty”.
  • Bandwidth – As with Option 2, don’t expect to migrate terabytes of data over the Internet using this method.  People with that kind of volume still need to look at Option 1 or a hybrid approach (see below).

The hybrid approach

One of the big limitations of Option 1 was the delays involved in populating the entire content database, shipping it to Microsoft, and getting them to plug it in for you.  But Options 2 and 3 were not effective ways to move huge volumes of data due to bandwidth issues.  The answer to resolving such a situation is the hybrid approach: do a little of both.

The basic idea is to bulk migrate the majority of the content over to your new environment well in advance of shutting off the old environment.  This is sometimes referred to as “pre-staging”.  You can take all the time you need to make sure things are migrated correctly and then promote the content databases to the hosted production environment.  Just before you are about to “go live”, perform a direct incremental migration of just the documents that were added or modified since the bulk migration occurred.

Using this approach, most of the content (say 99% of it) would be bulk migrated using Option 1 and only a small amount would be direct migrated via Option 2 or 3.  In addition, you would still have the ability to do direct migration if needed to deal with exceptions, make corrections, finalize links, etc.

But wait, there’s more…

There are still a few related topics to be discussed here.  I plan to get to them in upcoming articles:

  • Mapping your Notes users to Active Directory accounts that will work in your migrated environment.
  • How to preserve your Notes DocLinks in a migration to the cloud.
  • Deploying mixed SharePoint Online and on-premises SharePoint environments.

2 responses to “Migrating to the SharePoint Online platform (BPOS)

  1. Pingback: Getting Started with Migration via Native SharePoint 2010 Web Services « Notes SharePoint Blog

  2. Paul Beck August 3, 2011 at 12:11 pm

    Nice article, thanks for the BPOS-D info & quests tool can migrate Lotus Notes App DB’s