Notes SharePoint Blog

Steve Walch's blog about his favorite migration tool and other things related to Lotus Notes migration projects

Lotusphere bound

I will be heading to Lotusphere next week for the first time in a couple years.  It should be a blast to see the many familiar faces once again.  Let me know if you plan to be there and would like to meet up in person.

Notes Migrator for SharePoint Partner Training Webinar Series – Recordings now available!

My seven week Partner Training webcast series is now complete!  This is the most in-depth training for Notes Migrator for SharePoint delivered to-date and includes coverage of new 6.1 features as well as most existing features.  The 12 hours of riveting content have been recorded in HD and it is all free for you to watch now.  Use the links below to download recordings and slides or link to the You Tube versions.   
Session 1:  Migration Basics with the NMSP Designer Client
[view recording]

  • NMSP overview
  • Migrating standard template databases
  • Migrating custom database content
  • Managing document links
  • Mapping security permissions
  • User/group mapping options
  • Transforming data with formulas
  • Migrating to Wiki pages
  • Rendering and archiving documents

Session 2: Analysis and Automation with the NMSP Migration Console
[view recording]

  • First time Configuration Wizard
  • Database discovery
  • Usage analysis, data analysis
  • Introduction to design analysis and application complexity
  • Classifying and consolidating applications
  • Utilizing console views and reports
  • Exporting data for external reporting
  • Importing data into the repository
  • Provisioning SharePoint sites
  • Running migration jobs from the console
  • Automated provisioning and migration

Session 3: Advanced Content Migration Jobs
[view recording]

  • Overriding Document Metadata
  • Working with Lookups and Managed Metadata
  • Migrating images/attachments to alternate Libraries
  • Dealing with multi-valued items
  • Normalizing multi-valued items to multiple documents
  • Working with Content Types
  • Dealing with Response documents
  • Generating Microsoft Word and PDF documents
  • Setting links to parent documents

 Session 4: Tool Deployment, Domino.Doc and QuickPlace/QuickR migrations
[view recording]

  • Understanding tool deployment options
  • Migrating via SharePoint 2010 Web Services
  • Migrating via the NMSP Import Service
  • Installing and configuring the Import Service
  • QuickPlace/QuickR migration jobs
  • Automated QuickPlace/QuickR migrations
  • Domino.Doc migration jobs
  • Automated Domino.Doc migrations

 Session 5: Migrating Application Designs 
[view recording]

  • Overview of design migration capabilities (and limitations)
  • Migrating Notes schema to custom lists
  • Migrating Notes schema to Content Types
  • Advanced SharePoint column provisioning
  • Migrating Notes forms to InfoPath list forms
  • Migrating Notes forms to InfoPath form libraries
  • Migrating Notes views to SharePoint views

Session 6: Tips and Tricks for Managing Large Migration Projects
[view recording]

  • Design analysis revisited
  • Capturing copies of application designs during analysis
  • Blending tool and manual analysis
  • Rethinking application complexity
  • Triaging value of applications
  • Working with business owners
  • Managing the migration process
  • Managing doc links
  • Using Intermediate files
  • Migrating with the NMSPCMD.EXE command line 

Session 7: Even More Advanced Content Migration Jobs
[view recording]

  • Working with Document Sets
  • Migrating to Wiki and Basic Pages
  • Migrating to Web Part Pages
  • Migrating to Publishing Pages
  • Migrating to InfoPath Form Libraries
  • Introduction to SQL Server Migrations

How to send a sample Notes document to Quest support

If you contact Quest support with a migration job problem, they are likely to ask you for the migration job, a log file, and a sample Notes database.  Sending the sample Notes database is often problematic because (1) it is on the server, (2) it is too big to send by e-mail, (3) contains proprietary data, or (4) contains proprietary design information.  It turns out that for many issues, we do not need to see the entire database.  Most content migration problems can be debugged by looking at just one or two data documents and ignoring the database design completely. 

Here is a walkthrough of how to create a small empty Notes database with just one document in it.

1. In the Notes client, select the Notes documents you want to send to Quest and copy them to the clipboard (Ctrl-C).

image

2. In the Notes client, create a new database (Ctrl-N).  Specify a title and a file location on your local machine.  (You may want to specify your Desktop or some other folder you can easily locate later.)  Specify that you want to use the –Blank– template, and double check that you are not locally encrypting the database. 

image

3. When the new database opens up, paste the documents you copied in step 1 (Ctrl-V).

image

Note that the default view in the “blank” database will not show any columns (except the document number) and you will not be able to open the documents here.  But the data documents will be there and that should be enough for Quest to test the migration job.  If you want to, you can verify that you got the right documents using Document Properties (Alt-Enter).

image

4. Make sure that the database will be readable by Quest support by going to Database -> Access Control (Notes version 7 or earlier) or Application -> Access Control (Notes 8 or later). On the Basics tab, select the –Default– user and make sure that the Access level is set to “Manager”. On the Advanced tab, make sure that “Enforce a consistent ACL” option is NOT checked.

image

image

5. Close the database in the Notes client and locate the NSF database in the file system (in the folder you specified in step 2). This database should be very small (512 K plus the size of the document you copied) and you can probably email it “as is” or you can encode it inside a ZIP file.

Webcast recording on Migrating Notes Applications to SharePoint with Fewer Development Resources

Last week I recorded a webcast with SharePoint Technical Strategist Ira Fuchs from Microsoft.  We talked about fully leveraging the awesome “declarative development” capabilities in SharePoint and InfoPath for rebuilding complex Notes applications.  

75% of the presentation was about the concepts in Ira’s book for “Creating Enterprise Class Applications in SharePoint without Code”.  The rest was me trying to add the Notes migration perspective.  I thought the combination was very good and we had some good Q&A.

The webcast recording is here: http://www.quest.com/webcast-ondemand/-migrating-notes-applications-to-sharepoint-with-fewer-development-res816018.aspx

PS:  If you want to get a copy of Ira’s book for yourself, go to www.ihfpublishing.com and use coupon code “QUESTDISC” for a 30% discount.

image

Introducing the Notes Migrator for SharePoint 6.1 beta program

Notes Migrator for SharePoint 6.1 is a significant product release that pushes the product further in three important areas:  Design Migration, Content Migration and Pre-Migration Analysis. 

Design Migration

  • Migrate Notes Views to SharePoint Views
  • Content Type Generation
  • New Column Provisioning Options
  • Provision Calculated columns
  • Provision Lookup Fields

Content Migration

  • Migrate by Form Used
  • Improved Content Type Support in Migration Jobs
  • Normalize multi-valued items to multiple documents
  • Dynamic link tracking for URL columns
  • New Run Job options
  • Provision Alternate Libraries

Analysis / Migration Console

  • Capture Design Copies during Analysis
  • Classify by Last Used (All Replicas)
  • Import data into repository from CSV files

General

  • Performance and scalability for Migration Console
  • New User/Group Mapping options
  • Improved CBA/FBA support
  • Windows Authentication using alternate account for Link Tracking
  • Bulk Editing of certain database / class / job properties
  • Usability improvements

 

Once again, the beta will be managed on the new SharePoint for All community site.  If you would like to participate in the beta program, go to http://communities.quest.com/groups/notes-migration-product-beta-group.  Sign in with your Quest Community ID, or register to create a new one.  Then press the “Ask To Join This Group” button.

image

One of the site owners will review your request and will typically approve it the same day.  You will receive a notification and then get full access to technical content and (of course) the beta build itself.

Quest Support will not be able to help you with this version until it releases, so please use the group’s Discussion area for any questions, problems or suggestions.

Meet me at the SharePoint Conference

I will be at the Microsoft SharePoint Conference (SPC) in Anaheim, CA October 3-6.  Quest is the premier sponsor for the show and I will be spending a lot of time at the booth.

Our booth will have a kiosk dedicated to Notes to SharePoint migrations so please come by and see all that we have accomplished over the last year. 

I am also available for side meetings with customers throughout the event, so please contact steve.walch@quest.com if you would like to arrange something.  Or just stop by and say “hi”!

New Webcast: Migrating Lotus Notes Applications to SharePoint Online in Office 365

I recently co-presented a web presentation with Notes integration/migration rock star Gary Devendorf.  I was very happy with the results and the amount of “beef” we managed to squeeze into a marketing event.

See the recording here:  http://www.quest.com/events/ListDetails.aspx?ContentID=15382

image

How should I connect? How do I link? Which solutions do I need to install?

[Note: I am updating this old post to reflect the latest migration options in Notes Migrator for SharePoint 6.0.1.  Specifically, the "Lightweight Migration Service" is no longer needed.]

Connection options

Notes Migrator for SharePoint 6.0 now supports two very different ways to connect to SharePoint sites in order to migrate content to them. 

1. Quest Import Service.  The “classic” way is to install the Notes Migrator for SharePoint Import Service.  This is a stand-alone IIS web application that you run on one or more of your SharePoint front-end server boxes.  You have to directly access (or remote into) a SharePoint front-end server, run the NotesMigratorForSharePoint-Services-64bit-6.0.0.x.msi setup program, and select the “Import Service” option.  You need to be a farm administrator to install it and think about service accounts, permissions, etc. 

You also have to configure every new SharePoint site collection you create to use a particular Import Service instance.  To make this possible you also need to install the “Front End Services” solution included in the same NotesMigratorForSharePoint-Services-64bit-6.0.0.x.msi setup program.  Unless you are putting these components on different physical machines, you would simply install both components at once, which is the default. 

The Quest Import Service is definitely not trivial to install, but it is by far the most powerful and best performing option.  It should be noted, however, that there are three cases where the Quest Import Service cannot be used at all:

  • You are migrating to Office 365 (SharePoint Online Standard or Dedicated)
  • Your SharePoint site is using Claims Based Authentication
  • Your administrator refuses to install third-party code on your SharePoint environment

image

2. SharePoint 2010 Web Services.  The new option, for 2010 customers only, is to migrate via Microsoft’s new out-of-the-box web service.  This is much simpler to deploy – in fact there is often no need to deploy anything on your servers at all (see below).  There are really only two disadvantages to using this approach.  First, it can be significantly slower than running migrations via the Import Service.  Second, there is a slight limitation to how our Link Tracking Service works.  As you will see below, everything works in the end, but the user experience suffers a little until you finalize your links.

Linking options

Related your choice of connection options is the choice of Link Tracking Service options.  The Quest Link Tracking Service is an optional feature that keeps track of all the Notes documents you have migrated and dynamically redirects users to the current location.  I won’t go into all the details of the service here, but I want to focus on how the Link Redirector page works. 

If you enable the Link Tracking Service, every Notes DocLink (or HTTP link to a web enabled Notes document) in every migrated document gets converted to an HTTP link to a Link Redirector page (QuestLinkTracking.aspx).  This redirector page typically performs a lookup in a centralized Link Tracking database and then dynamically redirects the user to another migrated document in SharePoint (if it has been migrated) or to the original Notes version (if it has not yet been migrated).  So the natural question here is: Where does this Link Redirector page live and how does it get installed?

There are actually now two different versions of the Link Redirector page that you can choose from.  First is the “classic” one that you get when you install the Front End Services solution described above.  This one is configured on a per site collection basis, alongside the Quest Import Service.

An alternative version is the Sandbox Link Redirector page.  This version is intended for cases where you do not have the ability to install custom solutions and/or you cannot establish SQL connections from your server to the a shared Link Tracking database.  The main case we were thinking of when we designed this solution is Microsoft’s Office 365 environment and other highly secured hosting environments, but there will be plenty of people who prefer this option even for on-premises environments.  This page is packaged as a simple SharePoint solution (Quest.SandboxLinkRedirector.wsp).  Because it is a sandbox safe solution, it can actually be installed by any site collection administrator, even on locked down environments such as Office 365, without involving your farm administrators at all.

image

Note that because Sandbox Safe Link Redirector page does not connect to an external Link Tracking database, it always offers to redirect user to Notes, even if the document had been migrated to SharePoint.  In this scenario, users will not actually get redirected to their new SharePoint documents until their links are Finalized.  The saving grace here is that you can Finalize your links as often as you want to.  In productions migrations, customers often choose to Finalize links on a daily basis.

image

Putting it all together

Wow that is a lot of options and choices here!  Let me try to simplify things with a nice table.

Migration mode Quest Import Service SharePoint 2010 Web Services
SharePoint versions 2007, 2010 2010
Office 365 (BPOS) Dedicated only Dedicated and Standard
Server installation Administrator must run MSI, etc. None
Server configuration Per site collection None
Performance Fastest Slowest
Functional limitations None None
Link Tracking Service Full Dynamic Link Redirection (via Front End Services solution) Limited Redirection (via Sandbox Link Redirector solution) *

* NOTE: Strictly speaking, it is possible to install the Front End Services solution (with full dynamic link redirection using a Link Tracking database) even if you are not installing the Quest Import Service.  We believe, however, that most people will either want to install the full solution or keep things as light as possible and will not often mix and match.

Migration basics: Migrating to Document Libraries

From the perspective of a migration specialist, a library is very similar to a list. The main difference is that in a library, each “document” is an actual binary file with various data properties associated with it.

Therefore, migrating a Notes database to a SharePoint document library could be as simple as extracting binary file attachments out of each Notes document and placing them in the library.  This simplistic approach makes sense if the Notes application itself was designed to manage binary files—that is, if each Notes document is really just a wrapper around a binary file attachment.  Domino.Doc is an example of this type of application.  In the screen shot below Notes Migrator for SharePoint was used to extracted the attachments from each Notes document and placed them in a document library.  You can also extract various metadata items about each document and map them to SharePoint properties.

image

Be aware that several things can go wrong with this type of migration job. If there are no attachments in a particular Notes document (i.e., if it is just a normal rich text document), then nothing will be migrated to the library.  If there are multiple attachments in a particular Notes document, they may all be migrated to the library but they will no longer be one self-contained document.  In either case, you have probably misinterpreted the way the Notes application was used, and it should not have been migrated to a library in this manner. 

There are two answers to this dilemma.  The first possibility is to migrate the application to a list instead if a document library.  Even if it was called a “document library” in Notes, it may be more appropriate to map it to a custom list in SharePoint.

The second possibility is generate new documents (one for every Notes document) and check those into the document library.  Notes Migrator for SharePoint can convert Notes documents to a variety of file formats, including HTML, MIME, Word, PDF, and InfoPath.  These are the binary files that you check into user will open when they click on “the document”.   The three most popular choices for formatting Notes documents as binary files are discussed below.

Microsoft Word

Microsoft Word is a popular choice in environments that have standardized on Microsoft Office for document creation. The integration between Microsoft Office and SharePoint libraries is very good and can enable you to build a variety of powerful applications. Users can open documents from libraries, edit them and seamlessly save them back again. If a version control, check-in/check-out, or approval process or workflow has been enabled for the library, it will all work automatically. Office clients even support single sign-on with SharePoint and SharePoint Online Dedicated.

Office documents in SharePoint libraries are easy to search and you can even generate an instant SharePoint workspace to enable teams to collaborate on a particular document. Multiple users can open the same Word document and edit it at the same time, and users can see the changes being made by other users almost instantly.

When migrating Notes documents to the Microsoft Word format, you can migrate to simple unadorned Word documents or to custom Word templates. You can also migrate Notes data items to Word document properties or even to content controls in your custom template.

The following screen shot shows a Notes rich text document that was converted to a Word document on a custom letterhead template and checked into a SharePoint document library:

image

For more details on migrating to Word documents, see these posts.

PDF

PDF is another popular choice for migrating rich text Notes documents. Many organizations, especially in Europe, use PDFs to archive old content. Since PDF is now an open standard, the assumption is that there will always be a PDF reader such as Adobe Acrobat available in the future. An organization that has a large number of Notes databases with rich text documents may find that PDF is an ideal target format for many of them.

When PDF documents are placed in a SharePoint library, the integration is not quite as tight as it is with Office applications, but the user experience is still reasonable. Even though PDF readers and editors are not generally “SharePoint aware,” the experience of opening PDF documents is similar to downloading them from any web site. PDF documents can work with SharePoint’s search features, but you need to install a free add-on from Adobe for SharePoint’s full-text search indexer to read the content.

A word of warning about migration tools: a number of tools advertise the ability to convert Notes documents to PDF documents, but deliver poor results. If you plan to use this feature, we strongly recommend that you test the tools with your user’s most complex documents. Watch for how nested tables, embedded images, links to attachments and doc links are handled.

The following figure shows a Notes rich text document that was converted to a PDF document and checked into a SharePoint document library:

image

PDF migrations are discussed in more detail here.

Document rendering

In some cases, simply converting the rich text bodies of your Notes documents to Word or PDF files is not good enough because it does not capture the rich form layout that Notes users are used to. Without the form layout, you really haven’t captured all the information.

This is where an advanced concept known as document rendering comes in. With this technique, the migration tool “renders” each document with its original Notes form to generate a new rich text document that includes the entire form layout you had in Notes. To visualize this, compare the generated PDF document in the following figure with the PDF example provided above:

image

In addition to the rich text body, we captured the information that was presented with the original Notes form. Most significantly, we did not have to redevelop the form in SharePoint to accomplish this, nor did we have to explicitly map all the data fields that are displayed in the form header.

We used PDF in this example, but the same concept of rendering documents with forms applies equally well to Word documents, InfoPath documents, pages and even lists.

The Render With Form feature is discussed more fully here.

InfoPath

People sometimes choose the InfoPath document format when they want to migrate complex Notes applications to SharePoint, usually for one of two reasons: either the applications have complex data structures that do not lend themselves to being stored in a SharePoint List, or the applications have complex form designs that contain dynamically hidden sections, input validation rules, buttons, form events, or other sophisticated form logic. Ways of addressing the second issue are discussed in the section “Migrating Application Designs” below, so for now, we will focus on the migration of Notes content to InfoPath documents.

InfoPath data documents (traditionally called “InfoPath forms”) are really XML files that you edit using the InfoPath client (part of Office) or perhaps in a browser if your SharePoint server is running InfoPath Form Services. You specify the layout and behavior of your InfoPath form (and associate it with your desired XML schema) by creating an InfoPath form template. Typically you would store your XML data documents in a special type of SharePoint library known as a form library. This library is associated with one or more form templates in such a way that end users get a fairly seamless experience of creating, viewing and editing complex documents right from the library.

When performing a Notes migration, your main job is to convert Notes documents to InfoPath XML data documents according to a particular XML schema and check them into a form library.  Notes Migrator for SharePoint makes this fairly straightforward.  All you need to do is load in an InfoPath form template and specify how you want various Notes data elements to map to the various parts of your new XML schema.  These elements might include not only simple data fields, but also rich text, embedded images and attachments, links to external attachments, and links to other documents.  As XML schemas are not necessarily one dimensional, you may need to map one-to-many data items as well (for example, a product description document might contain multiple distributor names).

The following screen shot shows a Notes rich text document that was converted to an InfoPath document (associated with a specific InfoPath form template) and checked into a SharePoint form library:

image

NOTE:  We did not specify how the InfoPath form template was created. It may have been created from scratch by your InfoPath developer or you may have migrated an existing Notes form using Notes Migrator for SharePoint.

For more details on InfoPath migrations, look here.

Details, details, details

Additional considerations for migrating to SharePoint libraries include:

  • Notes documents contain metadata such as Created By, Created Date, Last Modified By, and Last Modified Date. Many migration tools drop this metadata during migrations to SharePoint, resulting in a major loss of business data.
  • Most Notes databases contain access control lists, which determine what specific users can do in a particular application. In addition, individual documents contain access restrictions such as Readers lists and Authors lists. Access definitions may use groups in the Domino Directory as well as roles defined for the database. Preserving all this information correctly in SharePoint may be critical to a successful migration of sensitive data.
  • If your Notes application has a concept of document versioning, make sure your migration tool allows you to correctly map the versioning to SharePoint versioning. All versions of a given document in Notes should appear to be versions of the same document in SharePoint.
  • It is common to want to assign folders during a migration. You can dynamically generate folder names in SharePoint based on data extracted from Notes.
  • Instead of folders, you may want to use a powerful new SharePoint feature called Document Sets. This feature is discussed in more detail here
  • When generating documents in a document library, you may with to put images and attachments in a subfolder or even in a different library.  This technique is discussed here.
  • For complex applications with mixed content, may wish to assign Content Types to documents as you migrate them.
  • Lists, Libraries and Pages

    There are three basic ways to store content in SharePoint: lists, libraries, and pages. Each of these has a number of interesting variations, but it is important to understand the differences between these three fundamental types so you can best decide what you want to migrate to. Each type is described briefly here; subsequent posts will explain in detail how to migrate content to each from Lotus Notes applications.

    Lists

    Lists are similar to tables in a relational database. A list is a flat collection of data records (called items in SharePoint) with a fixed set of data fields (called columns). Each data column has a fixed name and type. For example, a customer list may have a Text column called “Customer Name,” a Date column called “First Purchase Date,” and potentially dozens of other columns. One particularly interesting column type is Rich Text (also known as a “Note” “Body” column); this is where one would typically store large amounts of rich text. Lists can also have one or more binary attachments and may have one or more views, which allow users to select and sort the items in various ways.

    All of this should sound pretty familiar to Notes customers, because a list is actually the closest thing in SharePoint to a Notes database. The biggest difference is that SharePoint lists are highly structured with a fixed schema (like a relational database), whereas Notes databases can be very unstructured, with every document having a different set of data items.

    Libraries

    Libraries are collections of binary files, such as images, Word documents, or audio clips. While lists and libraries are very similar internally, the metaphor is very different: in a list, the document may contain several binary file attachments; in a library, the binary file is the document. The emphasis in libraries is the document management functionality, including versioning and check-in/check-out. As with lists, libraries can have many additional data columns defined for capturing additional information about each document.

    In the Notes world, the closest thing to a SharePoint library is a Domino.Doc file cabinet. (Domino.Doc was a popular document management system built on top of Notes.) Many organizations also built custom Notes applications that attempt to implement document management functionality. Any time you see a Notes application where the file attachment is “the document,” consider migrating it to a SharePoint library. It is also common for Notes “team site” applications to have a document library section as part of the overall application.

    Pages

    Pages are the building blocks of all SharePoint sites. These are the web pages you actually see in the web browser every time you click on a link to view a site, open a document, enter some information, or do just about anything else. Most people do not realize that the same pages that make up the sites themselves can also be used as data documents. SharePoint actually allows you to create several types of content pages, including basic pages, wiki pages, web part pages, and publishing pages. SharePoint Online Dedicated includes several nice publishing site templates that are designed to manage the authoring, approval and display of rich text web pages.

    While content pages have no exact equivalent in the Notes world, they can be a great way to migrate certain types of Notes applications. Any time you see a Notes application in which the main intent was to publish a library of rich text pages to a large number of users, consider migrating it to a SharePoint page library or a publishing site. This includes the many Notes applications that implemented public web or extranet sites.

    Follow

    Get every new post delivered to your Inbox.