Lotus Notes to SharePoint Blog

Blog about Dell's Notes Migrator to SharePoint tool and other things related to Lotus Notes migration projects

Category Archives: Tech Notes

Important notice regarding SharePoint 2013 support in Notes Migrator for SharePoint

When we shipped Notes Migrator for SharePoint 6.2 last month, this included support for SharePoint 2013 but we realized that had one very unfortunate limitation: 

The Import Service did not support SharePoint sites that used Claims Based Authentication.  This was actually not a new limitation, and it impacted a handful of SharePoint 2010 customers, but it suddenly became very important in SharePoint 2013.  The reason is that all SharePoint 2013 sites use claims by default.  So this effectively meant that most SharePoint 2013 users would be forced to migrate via the native SharePoint web services.  Not a terrible option, but definitely not as fast.  (Migrating via the Import Service can be 3 to 5 times as fast, especially if you have large attachments and use the Share Files Folder feature.) 

Happily, we have now resolved the limitation.  Starting with hotfix build 6.2.0.1060, The Quest Import Service will work as expected even with the new SharePoint 2013 defaults.  This fix will appear on our web site in our next point release (6.2.1), but many customers will not want to wait for that.  Therefore I would encourage anyone who wants to use the Import Service on SharePoint 2013 to proactively contact Quest support and request build 6.2.0.1060 (or any hotfix build after that).   

Direct Folder / Document Set Migration

In previous versions of Notes Migrator for SharePoint, users could map document metadata (for example the Category property of a Notes application or the {BinderName} property for Domino.Doc documents) that would cause folders to be created in SharePoint. Folders would be created as needed as documents were being migrated. This worked for most cases, but there were limitations that customers would occasionally ask about:

  • Since we only migrate folders as a “side effect” of migrating documents, there was no way to migrate empty folders.
  • Similarly, there was no way to create the folders ahead of time (before migrating the documents)
  • There was no way to set permissions, created/modified metadata, or additional data columns on the newly created folders

Now in Notes Migrator for SharePoint 6.2 we offer a way to do direct folder migration. This is really two separate features that work together…

Migrating records to folders

image  image

On the Advanced tab of your Target Data Definition, you can now indicate that you want to migrate to a folder in your target list or library.  In this mode of operation, every record you extract from the data source will result in a folder being created, instead of a document!  The only additional requirement is that you map at least one item to a target column of type Folder (which controls the new folder names). Many of the usual document migration features will now apply to folders including:

  • Mapping of permissions (using the “Map Reader/Author fields” checkbox on your Advanced tab)
  • Mapping created/modified metadata to folders (using the “Preserve Created/Modified” checkboxes on the Map Data tab)
  • Mapping additional data items to folders (requires creating a new Folder content type). 

Note that many job features that would apply to document migration will not apply to folder migrations. For example, document generation and duplicate document handling options would be disallowed in in this context.

Extracting information from Domino.Doc Binders

image image

One of the things that customers clearly want to do with this new feature is to migrate all the information from their Domino.doc Binders to SharePoint folders. To support this, we have added a new option to do exactly that in Domino.Doc Source Data Definitions. Simply check the Binders radio button on the Document Selection job, and now you are extracting Binders instead of Documents. Each row in the Preview represents a Binder in the current file cabinet and we have included additional columns for all of the standard Binder metadata available in Domino.Doc. Of course you can add additional columns to this query as well.

So putting these features together, you would typically map the {Title} property of your data source to a Folder column in your target. Simply checking “Map Reader/Author fields”, “Preserve Created/Modified identities”, and “Preserve Created/Modified dates” should bring over most of the other metadata but you can certainly add additional mappings if desired.

Note that this feature will only write new SharePoint folders; it will not update existing ones with the same name. So a best practice is to run the Binder migration job first (to create the folders with all the properties intact) and then run you normal document migration job.

Also note that we are planning a similar feature for extracting QuickPlace and QuickR folders, but this is not available in the current release.

Migrating to Document Sets

Similar to migrating to folders, Notes Migrator for SharePoint 6.2 also give you the ability to migrate directly to document sets. The situation here is very similar to what was described above. The tool already allowed creation of document sets as files within those document sets were being migrated. This is a powerful and popular feature, but suffered some of the same limitations.

  • Since we only create document sets as a “side effect” of migrating documents, there was no way to create empty document sets.
  • Similarly, there was no way to create the document sets ahead of time (before migrating the documents)
  • There was no way to set permissions or created/modified metadata on the newly created document sets separately from the documents.

The solution is similar to the folder solution described above. On the Advanced tab of your Target Data Definition, you can now indicate that you want to migrate to a document set in your target list or library.  In this mode of operation, every record you extract from the data source will result in a new empty document set being created, instead of a document!  The only additional requirement is that you add a target column of type DocumentSet and map a value to the DocumentSet.Name property. All of the other features of document set migration (described here) still apply. The difference is that every record you select gets mapped to a document set instead of a file within a document set.

Automatically ZIP file attachments

Many customers have requested the ability to compress Notes file attachments while migrating them to SharePoint. There are a number of good reasons for wanting to do this:

  • Save disk space on SharePoint server
  • Get around SharePoint file restrictions (i.e., blocked file extensions and/or size limits)
  • Improve the bandwidth sending data to remote SharePoint servers
  • Eliminate problems (hangs and memory leaks) when embedding certain types of file attachments inside Word documents

Notes Migrator for SharePoint 6.2 now allows you to achieve this in Notes and Domino.Doc migration jobs via a new property on Attachment columns in your Source Data Definition. The “Compress” property may be set to “None” (the default) or to “Zip”.

clip_image002

You can also configure a set of global exceptions to this rule on the Notes tab of the tool’s Options dialog. The “Compression Exclusions” option allows you to specify any file extensions that should never be zipped. This would typically include media files that are already well-compressed and would not benefit from zipping.

clip_image004

When the Compression property is set to “Zip”, all extracted attachments will be compressed and placed inside a ZIP file when migrated. As shown below, any icons or text links to the attachments will appear the same as before, but when the user clicks on them they will open up a ZIP file instead of the “raw” file.

clip_image006

Additional Notes

Zip files will always be excluded from further zipping, even if they are not specified in the “Compression Exclusions” list.

If you wish to use this feature to migrate files that were previously blocked by SharePoint (for example .EXE files), be sure to remove these extensions from your Blocked Files list on the SharePoint tab of the tool’s Options dialog.

When using this feature with Domino.Doc sources, add a custom Attachment column as described above. Map this custom column (instead of the predefined {Attachments} column) when mapping attachments to a target column.

clip_image008

This feature is not currently available to QuickR or QuickPlace.

Attachments in generated Word documents… use caution!

We have had a number of support cases on this over the last year, and we have made an effort to clarify our position on this in our latest release notes:

Migrating file attachments inside of MS Word documents is not recommended for large migration jobs. Since Word attachments are implemented as “Packager” OLE objects, the migration process is forced to invoke Packager code as well as the OLE handlers for the file type in question (often in separate processes). The problem is that each type of attachment is handled differently depending on which type of application created the attachment. So (for example) the first 1000 attachments may work fine and then document 1001 has a different type of attachment that causes a memory leak when Microsoft converts it into a Packager object. When migrating attachments and OLE Objects to embedded objects in MS Word, it is highly recommended that the workstation performing the migration has native applications installed that can open and edit every type of attachment that the migration jobs will be encountering during the migration.

The two recommended workarounds are:

  • Migrate attachments separately to the SharePoint document library. (The links from the Word documents to the attachments will be preserved.)

  • Place all attachments inside ZIP files inside the Word documents. (This is a new feature for version 6.2.)

If you still insist on embedding attachments into Word documents, you should understand that you are doing so at your own risk and that Quest support will probably not be able to help you if problems occur. Here are some tips that have helped other customers in the past:

  • There are known issues with packaging Adobe Acrobat 10 objects. If you have PDF attachments, be sure to install Adobe Acrobat Reader version 9 on the workstation.

  • Packager leaks usually have to do with cases where packager cannot use type-specific COM objects (including all .MSG files, and including .PDF files when the PDF reader is not installed)

  • Leaked resource handles accumulate per Windows process. This means that you should restart NMSP Designer between large jobs, even if they are successful. It also means that batches of jobs in the NMSP Migration Console would still not be a good idea.

  • There may other issues with packaging certain types of objects. Since we cannot product how third-party OLE handler will operate, this is beyond our control.

Introducing Notes Migrator for SharePoint 6.2

This release will be downloadable from the web site in a couple days. 

The biggest feature by far is full SharePoint 2013 support.  Everything you could do in SharePoint 2010 you can now do in SharePoint 2013 and the new version of Office 365.  Its that simple!

The other features from our release notes are:

  • Zip attachments while migrating
  • Support for Pass-through HTML
  • Special handling of documents containing just one attachment
  • Full folder migration (example: Domino.Doc binders)
  • Support for multiple Notes passwords
  • Improved lookup field support (add missing choices)
  • Improved support for migrating to Claims-based environments
  • Render with form improvements (computed subforms, etc.)
  • Discovery process is more robust

Some of these features will be more self-explanatory than others.  I will be posting detailed walkthroughs of a lot of these features here in coming days.  You can also check out the “what’s new” section of the user guide or watch my recorded “update” training here: http://communities.quest.com/docs/DOC-14812

Checklist for planning the migration of a Notes Database

Before migrating a Notes database to SharePoint, there are a variety of details to gather and choices to make. Thinking through the items on this checklist will help ensure a smooth migration process with minimal surprises.

Source database details

  • Source type (Notes, QuickPlace, QuickR, Domino.Doc)
  • Source name
  • Notes template used (if any)
    • Level of customization from template
  • Domino server & path (closest replica to migration team)
  • Special Notes identity required to access database (if any)
  • Public keys needed to decrypt documents (if any)
  • Approximate number of Notes documents to be migrated
  • Approximate size of Notes content to be migrated
  • Desired priority of migration
  • Desired phase/group of migration (intended to capture clusters of databases that should logically be migrated together)
  • Documents that should / should not be migrated
  • Unique “document types” to be migrated (this usually correlates to distinct Notes forms that back existing data documents)
  • Design Details that will need to be accounted for
    • Forms
    • Workflow
    • External Connectivity
  • Third Party validation/security auditing
    • PCI Compliance
    • FDA Compliance
    • HIPA
  • Security and permissions that will need to be maintained and updated
  • Archiving of data
    • Date Range
    • Data Type

Target Site Details

  • SharePoint site URL
  • Account to be used or provisioning and content migration
  • New or existing site?
    • Template to use for provisioning site (may be custom developed)
    • Site provisioning details – parent site, inherit permissions
    • Provision site security (user permissions, group permissions, roles)?
    • Are specific content databases required?
  • Source of identity mapping information
    • Users
    • Groups
  • Out-of-box SharePoint features to enable/leverage
  • Do file size limits and blocked file types need to be changed from the defaults
  • Site Columns in use
  • Content Types created
  • Workflow type and creation
  • Search and Indexing

Target List/Library (specify for each document type if appropriate)

  • Target list or library name (Naming schemes in use)
  • New or existing list?
    • Template to use for provisioning list (may be custom developed)
    • Add to Quick Launch
  • Provision list/library security (user permissions, group permissions, roles)?
  • Allow tool to upgrade existing schemas if needed?
  • Desired folder structure within list or library
  • Map created/modified identities/dates
    • Behavior if user mapping fails (fail or substitute default user)
  • Out-of-box SharePoint features to enable/leverage
  • Third Party solutions (templates, web parts, etc.) in use or available

Mapping details (specify for each document type if appropriate)

  • Content Type for migrated documents (if any)
  • Target document format
    • List Items
    • Extracted binary file attachments only
    • Generate InfoPath docs (which template; target Forms Server)
    • Generate Word 2007 docs (which template)
    • Generate Adobe Acrobat (PDF) documents
    • Generate Web Part pages (which template)
    • Generate Content pages – Publishing pages, Basic pages, Wiki pages
    • Generate HTML files
    • Generate MIME files
  • Use separate library/folder for attachments, images, embedded objects?
  • Field mappings
    • Source data item/column names and types
    • Data transform required (example: translation of keyword values)
    • Target data column names and types
    • Using Lookup columns?
    • Using Managed Metadata columns?
  • Migration options
    • Map document level security (readers / writers)?
    • Convert DocLinks to dynamic links (i.e. use the Link Tracking Service)?
    • Map discussion/response hierarchies
    • Map calendar logic (repeating meetings)
    • Map approval status / workflow state? (How expressed in Notes app?)
    • Map version histories (How expressed in Notes app?)
    • Map data in date ranges
    • Render with form (archive only)

 

NOTE: Two important items were not covered in the above checklist:

  • Development work that may be needed in SharePoint Designer, InfoPath, Visual Studio to reproduce the custom logic you had in Notes.
  • The opportunity to reduce your migration effort by consolidating similar application designs, reusing custom development work, and automating large scale migrations.

Both of these topics are discussed in detail in the Quest white paper “Migrating Lotus Notes Applications to Microsoft SharePoint: Understanding Application Complexity and the Value of Consolidation and Automation” which may be download from http://www.quest.com/documents/landing.aspx?id=9746.

Exporting attachments and generated files to the file system

In case you missed it:  in Session 6 of my Partner Training webcast series, I introduced a free utility for reorganizing the intermediate files that Notes Migrator for SharePoint writes to disk.  Not everyone was clear on what such a utility was good for, so I thought I should elaborate:

Every once in a while, we hear one of theses requirements from customers:

  • Extract all the attachments from a Notes database and place them in the file system (instead of SharePoint)
  • Convert Notes documents to Word Documents and place them in the file system (instead of SharePoint)
  • Convert Notes documents to PDF Documents and place them in the file system (instead of SharePoint)
  • Convert Notes documents to InfoPath Documents and place them in the file system (instead of SharePoint)

The trick to doing any of these things is to use our intermediate data file format.  Just design a Notes Migrator for SharePoint migration job and when running the job, select Save to Intermediate File.  This will give you the option of saving any attachments or generated documents as disk file rather than encoding them in the main XML output file.

image

The problem with this approach is that the files that are saved to disk are not really that useful because they are arranged by Notes UNID, not any way that would be useful to humans.  That’s where the free utility come in.  This utility, which may be downloaded from our Notes migration community site, will take all these files and place them in the folder (or network share) of your choice and arrange them according to your requirements.  If you want to see an example of this tool in action, watch the video of Session 6.  Slides starting at about minute 77 and demo starting at minute 91.  Have fun!

Notes Migrator for SharePoint 6.1 Reviewers Guide

Notes Migrator for SharePoint 6.1 is a significant product release that pushes the product further in three important areas:  Design Migration, Content Migration and Pre-Migration Analysis.  This release will be “Generally Available” on the Quest web site in a few days.  If you were in the beta program you already have the final build (6.1.0.626).  Below is a high level list of this release’s most important new features.

Design Migration

Migrate Notes Views to SharePoint Views – This feature will allow you to select a Notes view and generate a similar view in any SharePoint list or library.  Because the mapping of columns in your view is intimately tied to the mapping and provisioning of data columns that occurs in your migration job, this new capability is also tied to your migration job.  Specifically, view migration is found on the new Advanced tab of your Target Data Definition.  You can design new views there and the views are provisioned when the migration job is run.  A nice side effect of this model is that you can run the same migration job against many new or existing lists or libraries.

clip_image001

clip_image002

Press “Import from Notes” and then select the Notes view you want to migrate.  The view migration wizard will then attempt to design an equivalent SharePoint view, mapping as many Notes columns as possible to existing SharePoint columns OR to new columns defined in your job’s target data definition.  As you might expect, there are a number of things that can go wrong with such an automated mapping.  The two most likely issues are (1) the Notes data item shown in the view had not been migrated to the SharePoint list yet or (2) the Notes view column is computed and the formula is too complicated for a tool to convert.  For issues like this, manual intervention is required and the migration wizard guides you through that.  In the View Columns step, it gives you a side by side view of the Notes and SharePoint columns and highlights the parts that it needs help with in red.  You can decide to manually map a Notes existing columns, define a new data column (which will be added to your migration job), or even specify a formula for a new computed column in SharePoint.  Similarly you can also specify the sort order, the grouping and the document selection rules, or you can just take the defaults. 

Content Type Generation – These feature takes the tool’s ability to migrate schema (select fields from a custom Notes application and provision a similar schema in a custom list) to a new level.  Now you can migrate your custom Notes application schema to SharePoint content types instead.  You can create new Content Type Definitions on the new Content Types tab of your Database or Technical Class records in the Migration Console.  You can design them from scratch or, more likely, you will generate the Content Type definitions from a Notes Form, from an existing migration job, or from an existing list that you have been working on.  (The last two options allow you to do your initial development and testing on a custom list and then transition to content types later.)

clip_image003

clip_image004

Once you have generated an initial Content Type Definition, you can further customize it, specifying the parent content type, the group, and the columns.  The columns editor looks like the Target Data Definition columns editor, but in this case you are defining Site Columns instead of List columns.  When you are ready to provision your new Content Type, the tool will try to find existing Site Columns that match your specification and will provision new ones if needed. 

clip_image005

clip_image006

New Column Provisioning Options – The ability to provision lists and libraries while migrating content has long been a popular feature.  This capability has been greatly expanded to allow provisioning a nearly complete list of SharePoint column options, such as default values and data validation.  You can even set columns to be read-only (something you can’t normally do in the SharePoint user interface).  Note that as before, the provisioning options only apply when the tool is first provisioning the columns; it does not upgrade existing columns.

clip_image007

Provision Calculated columns You can now define Calculated columns in your target data definition.  These will be provisioned in SharePoint when your migration jobs are run. This Microsoft documentation page explains the legal syntax for SharePoint formulas:  http://msdn.microsoft.com/en-us/library/bb862071.aspx.  You can also use our new built-in formula editor, which appears at several points in the product.

clip_image008

clip_image010

Provision Lookup Fields – Lookup fields can now be provisioned in SharePoint lists.  Previous versions of the tool allowed you to migrate to Lookup fields, but you first had to manually configure them in SharePoint. You can even configure lookups from lists in other sites in the same site collection (something you can’t normally do in the SharePoint user interface).

clip_image011

Content Migration

New Run Job options – During normal migration jobs, Notes Migrator for SharePoint first takes care of the provisioning steps (creating the list/library if needed, adding the appropriate content types to the list, adding any missing columns if needed, and creating views, setting the list/library permissions) and then migrates the content.  Now you can now choose to run just the provisioning bits without migrating the content.  This will be useful in cases where you want to review the resulting list schema and perhaps make changes to the list settings before migrating content.

clip_image013

clip_image015

Also note that the above screen shots show off the tool’s new Run Job button with the drop down menu of various modes of running the tool, such as importing data to and importing data from intermediate XML files.  Since this button is also available in the Migration Console, users can now leverage the intermediate file options from there (it was formerly available in the Designer Client only).  Similar functionality is also available in the tool menus and the bulk Migrate To SharePoint task in the Migration Console.

Migrate by Form Used – When selecting records from your Notes database, you can now elect to query records by form.  You can pick one or more Forms from the Notes database design or type them manually.  This will limit the records selected to include only those data records that were flagged with the forms you selected (i.e., where the “Form” item was set to one of those form names).

clip_image017

clip_image019

If you specify forms in this way, the Select Items and Columns dialog will also display the data items defined on your selected forms.  (Remember, however, that background agents, etc., may have also set data items on the Notes documents, so you may still need to look at the Sampled Items node to discover those.)

Improved Content Type Support in Migration Jobs – Notes Migrator for SharePoint already had pretty good support for dealing with content types while migrating content, but this release extends those capabilities and also makes it much easier to use.  The new Manage Content Types section on the first tab of the Target Data Definition allows you to push site content types into a new or existing SharePoint list.  Adding the content type to the list is a prerequisite to writing documents with those content types, so this new feature eliminates that manual step and make more automation possible. 

clip_image021

In addition, you can now associate each content type with one or more Notes forms.  If you do this, then the tool will automatically assign the content type for each migrated document without the need to do any explicit mappings.  (For more complex scenarios you can still explicitly map any piece Notes data, including the result of a formula evaluation, to a ContentType field.)

Note that this feature is completely independent of the ability to generate new Site Content Types, as described above under Design Migration, but you may often find yourself using the two in combination.

Normalize multi-valued items to multiple documents – It is common for Notes forms to implement detail records (the things that relational developers would have set up a one-many-relationship for) using multi-valued data fields arranged to look like a table, as shown below. Users would then fill in as many entries as the needed, making sure that the values from each multi-valued item lined up properly.

clip_image023

clip_image025

Now you can use Notes Migrator for SharePoint to extract the multi-valued items as separate records. To do this set the Multi-Value disposition option for the columns that you expect to contain arrays (ContactName, ContactTitle, and ContactPhone in the above example) to “Normalize”. This will cause NMSP to generate multiple SharePoint items (three in the above example) for the values in the columns designated for Normalization.

clip_image027

clip_image029

Note that if you also wanted “header” records you could first migrate those using a separate migration job. You could use lookup fields or some other mechanism to relate the “header” and “detail” columns. In the above example, Customer ID in the Contacts list is set up as a lookup field that references the Customers list (where Customer Name and Category are also stored).

Finally, you may encounter Notes forms that list out each item in separate rows, instead of using multi-valued items. NMSP can handle that case as well, but the migration team will need to do a little work with formulas to generate multi-valued columns to Normalize on.  For example, you could define a source data definition column with the formula “Product_1: Product_2: Product_3: Product_4” to generate a “Products” array.

clip_image031

Dynamic link tracking for URL columns – When migrating Notes data to SharePoint URL fields, you can now leverage the tool’s popular Link Tracking Service for those links (the same way you currently do for Doc Links in Rich Text Fields).  You do not need to do anything special to enable this.  Simply start migrating data items that store notes:// links (or formulas that generate notes:// links) and they will be converted to dynamic links that will ultimately point users to the migrated versions of those documents. 

As a reminder, the tool supports two input formats for Url fields.

  • <url> –  sets the url to be the display name in SharePoint
  • <url>, < name> –  sets the url and display name separately in SharePoint

Improvements in “Render With Form” function – This very popular feature, which allows you to extract the content of any custom Notes database (regardless of the complexity) so you can archive them as simple rich text documents in SharePoint, has been improved in a number of respects. Handling of dynamic subforms, computed fields, computed text, and keyword fields have been improved, resulting in even better looking rendered documents.

Provision Alternate Libraries – Users have often relied on the tool’s unique ability to migrate images and attachments to a separate location from the main documents being migrated.  For example, you might want to place all attachments in the Shared Documents or Site Assets document library.  Previously you would have had to provision these alternate libraries manually before running your migration job.  Now the tool provisions these automatically if needed.

clip_image032

Analysis / Migration Console

Capture Design Copies during Analysis – Now when you perform a Design Analysis on a set of databases you can elect to also create a local design copy.  For every database scanned, the tool will create a small copy of the database (design elements only, similar to a design template) on the local machine on which the design analysis is being performed.  The intent of this feature is to allow for consultants and other migration personnel to be able to view the full database designs even while they are disconnected from the production environment, which is particularly important on large analysis projects where in-depth manual design analysis is required.  The location of these databases is controlled by a setting on the Analysis tab of your Global Options dialog, and defaults to a folder in the tool’s <ProgramData> area.  Users can the easily open the Local Design Copy from any database view.

clip_image033 clip_image034

Classify by Last Used (All Replicas) – This enhancement expands the available options for automatically determining which Technical Class or Business class a databases belongs to.  Now you create a rule for recognizing class members based on when it was last used across all known replicas.  For example, you can create a class that groups all databases with a Last Used date greater than 356 days ago.

clip_image036

Import data into repository from CSV files – This enhancement allows users to read in records from a CSV file and update vales in certain Quest Repository database records.  The tool is available on the main Notes Migrator for SharePoint node.  First the user is prompted to select a CSV file and then a mapping dialog is displayed.  For each column in the CSV file, the user can choose to map it to one of the available database record properties.  Not all database properties can be imported from external data sources.  The intention here is that users who perform manual analysis/triage projects using spreadsheets or external tools can import that data back into the Migration Console.

This feature can also be used to add new database records into the repository (by Server and File Path, Replica ID, or Database Key).  The intention here is to allow users to import a list of databases to be analyzed or migrated.  Users would typically follow up such an import with a full analysis of those databases to populate the remaining database properties.

clip_image038

General

Performance and scalability for Migration Console – The database views have been re-architected to support 60,000+ databases at a time.  With a very large repository, users may experience a delay while starting the tool as all the records are loaded into memory.  After that, scrolling, sorting and filtering database views and opening additional views should be very fast.  Special protections have also been added to prevent you from running very large reports that are likely to crash the migration console.

New User/Group Mapping options – When performing user/group name mapping, the Output Translation option now allows you to further transform the name that results from your name lookup before submitting that name for resolution in SharePoint.  This may be useful when your environment requires specific name formats that are not immediately available in your mapping source.

Also, the Test User/Group Mapping tool now has a Validate in SharePoint button which will try to resolve the name that your configured mapping process produces so you can verify that it really works in SharePoint. This should make experimenting with and debugging various user mapping options a little easier for everyone.  Note that this capability is only available when using client-side user mapping (not when configuring server-side mapping in the Import Service).

clip_image039

Improved CBA/FBA support – Connections to SharePoint sites using Forms Based Authentication now automatically renew cookies as needed during long migration jobs and other operations.  The tool will routinely check the expiration time of the authentication tokens that it holds for the client and, if a token is due expire within a certain window, it will force a new authentication.  (The time limit will be 30 minutes, but this can be changed in your SharePoint Environment settings.)  Depending on your particular authentication system, this may appear to the user as a browser prompt forcing the user to re-authenticate.  In other cases, it may appear as a browser window that opens briefly and then closes again.

clip_image040

Windows Authentication using alternate account for Link Tracking Database One new connection option is available for connection to the Link Tracking Database.  You can now specify that you want to use Windows Authentication, but supply an account other than your own.

clip_image041

Bulk Editing of certain database / class / job properties – Expanding on the tool’s capability to select multiple databases and set properties in bulk, you can now set a number of additional properties in a large number of selected databases.  You can even set certain properties inside the migration jobs assigned to the selected databases.  Finally, you now have a similar set of options for Technical Classes and Business Classes as well.

Usability improvements A number of things have changed in the product to improve to make the tool easier to use, including rearrangement of dialogs and additions of menus.  More pop-up help icons and more context sensitive links to help topics have been added.  A completely rebuilt “Add/Remove Columns” dialog and the built-in documentation describing the 170 possible built in columns makes customizing views much easier.

clip_image043

clip_image045

Suggested RFP / POC points for a Notes to SharePoint migration tool

There sure are a lot of organizations conducting Notes-to-SharePoint pilot projects, migration tool “bake offs”, POC projects, and RFPs these days!  I often get asked for my suggestions as to what tool criteria people should include in their formal evaluations.  It turns out that the details are very important; there are a number of tools out there that claim to offer some high-level “check box” features, but you need to dig a little to see if those features are really implemented adequately for your real-world migration projects.  So I thought I would share my ever-growing list of tool considerations here.

Note that this list is probably way more than anyone would want to include in any one evaluation project.  My suggestion is to take this as a list of possible items to consider and then select which points will be most important to your users and your particular applications. 

General

  • Tool performs complete pre-migration analysis
    • Tool can be installed and run without consultant intervention
    • Analysis functionality is fully integrated with the migration tool (flowing smoothly from pre-migration analysis, detailed analysis and application consolidation, project management, assignment of targets, design of migration jobs, automated provisioning and bulk migration based on analysis results)
    • Tool facilitates migration project costing with the ability to track estimates for individual applications as well as  consolidated application designs
  • Tool supports migration to SharePoint 2007 and 2010
    • Includes option to migrate over native SharePoint 2010 web services, without installing any vendor code on SharePoint server
    • Includes option to migrate over vendor’s web service optimized for high performance migrations
  • Tool supports migration to Office 365 (SharePoint Online)
    • Supports migration to Office 365 standard (multi-tenant)
    • Support migrating to Office 365 Dedicated via an on-premises staging environment
    • Supports direct migration to Office 365 Dedicated
  • Tool includes a Command Line utility for scripting migration
  • Tool supports migrating to SQL Server, including data normalization, rich text, images, attachments, doc links
  • Tool supports migrating to the file system
    • Supports generating XML files from Notes content
    • Supports exporting file attachments, etc. to file system
    • Supports generating files in the file system (HTML, Word, PDF)

Core Content Migration Details

  • Supports mapping Notes data to any SharePoint column
    • Migrate from Notes items
    • Transform data at migration time using formulas
  • Tool can migrate to SharePoint choice columns
    • Includes option to add missing values to column schema as needed
  • Tool can migrate Notes names to User columns
  • Tool can preserve Notes document metadata
    • Preserves Created/Modified dates
    • Preserves Created By/Modified By identities
  • Tool supports migrating Notes rich text with full fidelity
    • Preserves fonts, table, nested tables, bullets, margins, etc.
    • Preserves embedded images
    • Preserves embedded image resources
    • Converts embedded OLE objects to equivalent file type
    • Migrates embedded attachments
    • Migrates document level attachments
    • Preserves collapsible sections and section headers
  • Tool supports full provisioning of security constructs
    • Provides several options for mapping Notes names to SharePoint (AD) names
    • Maps ACL permissions to either the site level or the list/library level
    • Can set explicit permissions for SharePoint objects based on Notes ACL, including support for custom permission levels
    • Can add users to the existing site security groups based on Notes ACL, including custom site groups
    • Can provision and populate SharePoint Groups from Notes ACL roles
    • Can provision and populate SharePoint Groups from Domino directory groups
  • Tool supports incremental migration
    • Can rerun job, detecting new and changed documents
    • Can write modified documents as new SharePoint versions
  • Tool supports migrating Notes calendars to SharePoint calendars
    • Supports repeating meetings, all day events, etc.
  • Tool supports for migrating Notes discussions to SharePoint discussions lists
    • Supports threaded discussion
    • Preserves subject fields of response documents
    • Offers optional custom web part for presenting discussion lists with a more Notes-like user interface
  • Tool correctly handles Notes doc links and HTTP links
    • Preserve links between documents in different applications at different times
    • Links can work between all types of migration targets (lists, pages, etc.)
    • Intra-document HTTP links (in QuickPlace or other Domino web-enabled apps) are recognized as doc links and translated accordingly
    • Link tracking system that continuously manages doc links as content moves through staging, deployment and production
    • Link tracking system allows finalization (replacing dynamic links with permanent links) after the migration is complete (even on SharePoint Online)
    • If company also provides mail migration technology, are links preserved between migrated mailboxes in Exchange and migrated applications in SharePoint?
  • Tool can provision sites, list and libraries using custom SharePoint templates

Advanced Content Migration Details

  • Tool provides a sufficient choice of migration SharePoint migration targets to accommodate various business and end-user application requirements:
    • Migrate to List Items
    • Migrate files to Document Libraries
    • Generate generic Word documents
    • Generate Word documents using custom templates, properties and content controls
    • Generate PDF documents
    • Migrate to Wiki pages
    • Migrate to Basic pages
    • Migrate to Publishing pages (Other tools advertise advertises this but is very limited)
    • Migrate to Web Part pages, mapping Notes content to any property on any web part
  • Tool can render complex Notes Documents into rich text (for archiving purposes)
    • Tool can render entire Notes documents with their original Notes form to a single rich text column or generated rich text document
    • Tool can render entire Notes documents with a designated form to a single rich text column or generated rich text document
  • Tool supports mapping document approval/publishing status from Notes data to the out-of-box approval/publishing feature in SharePoint
  • Tool can migrate Notes content to SharePoint 2010 Managed Metadata columns
    • Support matching terms by alias or locale
    • Support matching terms by hierarchy
    • Supports adding missing terms to term store
  • Tool supports SharePoint 2010 Document Sets
    • Individual Notes document can be mapped to individual doc set (splitting out all the generate files, attachments, etc., as members of the doc set)
    • Multiple related Notes documents can be merged into a single doc set
  • Tool can support data normalization
    • Embedded images, attachments and objects can be normalized to separate SharePoint libraries or folders (keeping the various parts linked)
    • Documents with multi-valued items can be normalized into header and detail documents
    • Documents with tables of related items can be normalized into header and detail documents
  • Tool can dynamically set content types based on data in the Notes document
  • Tool can migrate to InfoPath form libraries
    • Generates documents according to any XML schema
    • Supports rich text, embedded images, attachments
    • Supports migrating multi-valued items to repeating XML elements
    • Supports user (people picker) fields

Design Migration Details

  • Tool supports extracting the schema of a Notes application and provisioning  a similar schema in SharePoint lists and libraries
    • Can determine schema based on Notes forms
    • Can determine schema based on sampled Notes documents
  • Tool supports extracting the schema of a Notes application and provisioning  a similar schema as SharePoint Content Types
    • Can also push these content types into specific lists and libraries as they are provisioned
  • Tool can provision new columns in new or existing lists and libraries
    • Including choice and lookup columns
    • Including data validation, required field, and other standard column options
  • Tool can generate “classic” InfoPath Document Form templates (for use in SharePoint form libraries)
    • Includes form layout for form and all mappable field types
    • Includes rich text and attachment areas
    • Includes option to generate InfoPath sections for area with hide-when logic
    • Includes special handling for Notes static subforms and computed subforms
    • Provides a detailed report itemizing whatever elements could not be migrated
  • Tool can generate InfoPath List Forms (new with SharePoint 2010) from existing Notes forms
  • Tool can provision SharePoint Views based on Notes views
    • Includes a migration wizard allowing users to define computed columns, etc., where needed
  • Tool includes support for consolidating applications with similar designs and automating their migration
    • Recognize class members using template names
    • Recognize class members using full design comparison

QuickR / QuickPlace Details

  • Tool automates the provisioning of SharePoint sites based on the QuickR / QuickPlace room hierarchy
    • Each sub-room is mapped to a SharePoint sub-site
    • Uses SharePoint Team templates or custom site templates
    • Preserves the access control settings of each room and sub-room
  • Tool migrates all types of QuickR / QuickPlace pages
    • Migrates all standard page types: Imported pages, Link pages, Discussion pages, Task pages
    • Migrates custom Page Types, including custom Fields
    • Offers migration of Published documents, Unpublished documents, or both
    • Offers migration of “orphan” documents that are accessible through navigation hierarchy
  • Tool correctly migrates rich text in QuickR / QuickPlace pages
    • Full rich text fidelity is preserved
    • Embedded images are retained
    • URL links that point to other QuickR / QuickPlace pages are converted to SharePoint links
    • Preserves document level permissions
  • Tool preserves QuickR / QuickPlace navigation
    • Converts place menus to SharePoint QuickLaunch area
  • Tool includes pre-migration analysis features that are QuickR / QuickPlace aware
    • Excludes configuration documents from data analysis
    • Summarizes data documents by Page Type used in each room
    • Includes configuration documents in design analysis
    • Detects rooms where custom Page Types and Fields have been defined by end users

Analysis Details

  • Tool performs application usage analysis
    • Includes exclusion of designated user names who should not count as real users
    • Can aggregate usage information across multiple replicas of each database
  • Tool performs Data analysis, finding patterns of actual usage in a particular database
    • Includes detection of blocked file attachment extensions and oversized files that will not be allowed on SharePoint
    • Includes analysis by Form Used, making it easy to extract the “document types” actually used in a database instance
  • Tool performs Design analysis,  summarizing design element details
    • Includes initial complexity calculations
  • Tool can compares all application designs to a predetermined set of templates, which assists in  consolidating applications with similar designs and automating their migration
  • Tool can capture design copies of each database while scanning the environment 
  • Tool can extract all users and group names contained in a set of databases (ACLs, document metadata, usage history, etc.) making it easy to proactively plan for how they will map to SharePoint
  • Data repository is an open format that customers are free to use in any way they see fit
  • All analysis details may be exported to external format such as CSV or XML for use with other reporting tools such as Microsoft Excel

Company

  • Describe track record of delivering top-notch support around the world
    • Support call centers located globally for 24×7 coverage
    • Many counties are staffed with presales and post-sales consultants
    • Company offers well-defined SLA’s including policies for escalating to development
    • Company frequently delivers feature enhancements to unblock large migration projects
  • Company has tools for moving content from SharePoint to SharePoint (upgrades, reorganization, deployment to production) and well as from other data sources to SharePoint
  • Company has related tools for managing SharePoint environments, including discovery, reporting, policy enforcement, and granular recovery
  • Company also has tools for performing Notes-to-Exchange migrations and directory migrations
  • Company offers a set of advanced web parts for reducing the time and cost of rebuilding complex custom applications
  • Company has an experienced professional services team to participate in and/or deliver migration projects to ensure project success
  • Company has an experienced partner network able to deliver migration projects.  For large Notes projects this should include partners with off-shore resources who’s teams utilize tool in a migration factory approach to deliver high-volume, low-cost migrations
  • Company has an established track record in this unique migration space.  Company is able to provide case studies, project references, best practices materials, facilitate POCs, etc.

How to send a sample Notes document to Quest support

If you contact Quest support with a migration job problem, they are likely to ask you for the migration job, a log file, and a sample Notes database.  Sending the sample Notes database is often problematic because (1) it is on the server, (2) it is too big to send by e-mail, (3) contains proprietary data, or (4) contains proprietary design information.  It turns out that for many issues, we do not need to see the entire database.  Most content migration problems can be debugged by looking at just one or two data documents and ignoring the database design completely. 

Here is a walkthrough of how to create a small empty Notes database with just one document in it.

1. In the Notes client, select the Notes documents you want to send to Quest and copy them to the clipboard (Ctrl-C).

image

2. In the Notes client, create a new database (Ctrl-N).  Specify a title and a file location on your local machine.  (You may want to specify your Desktop or some other folder you can easily locate later.)  Specify that you want to use the –Blank– template, and double check that you are not locally encrypting the database. 

image

3. When the new database opens up, paste the documents you copied in step 1 (Ctrl-V).

image

Note that the default view in the “blank” database will not show any columns (except the document number) and you will not be able to open the documents here.  But the data documents will be there and that should be enough for Quest to test the migration job.  If you want to, you can verify that you got the right documents using Document Properties (Alt-Enter).

image

4. Make sure that the database will be readable by Quest support by going to Database -> Access Control (Notes version 7 or earlier) or Application -> Access Control (Notes 8 or later). On the Basics tab, select the –Default– user and make sure that the Access level is set to “Manager”. On the Advanced tab, make sure that “Enforce a consistent ACL” option is NOT checked.

image

image

5. Close the database in the Notes client and locate the NSF database in the file system (in the folder you specified in step 2). This database should be very small (512 K plus the size of the document you copied) and you can probably email it “as is” or you can encode it inside a ZIP file.

How should I connect? How do I link? Which solutions do I need to install?

[Note: I am updating this old post to reflect the latest migration options in Notes Migrator for SharePoint 6.0.1.  Specifically, the “Lightweight Migration Service” is no longer needed.]

Connection options

Notes Migrator for SharePoint 6.0 now supports two very different ways to connect to SharePoint sites in order to migrate content to them. 

1. Quest Import Service.  The “classic” way is to install the Notes Migrator for SharePoint Import Service.  This is a stand-alone IIS web application that you run on one or more of your SharePoint front-end server boxes.  You have to directly access (or remote into) a SharePoint front-end server, run the NotesMigratorForSharePoint-Services-64bit-6.0.0.x.msi setup program, and select the “Import Service” option.  You need to be a farm administrator to install it and think about service accounts, permissions, etc. 

You also have to configure every new SharePoint site collection you create to use a particular Import Service instance.  To make this possible you also need to install the “Front End Services” solution included in the same NotesMigratorForSharePoint-Services-64bit-6.0.0.x.msi setup program.  Unless you are putting these components on different physical machines, you would simply install both components at once, which is the default. 

The Quest Import Service is definitely not trivial to install, but it is by far the most powerful and best performing option.  It should be noted, however, that there are three cases where the Quest Import Service cannot be used at all:

  • You are migrating to Office 365 (SharePoint Online Standard or Dedicated)
  • Your SharePoint site is using Claims Based Authentication
  • Your administrator refuses to install third-party code on your SharePoint environment

image

2. SharePoint 2010 Web Services.  The new option, for 2010 customers only, is to migrate via Microsoft’s new out-of-the-box web service.  This is much simpler to deploy – in fact there is often no need to deploy anything on your servers at all (see below).  There are really only two disadvantages to using this approach.  First, it can be significantly slower than running migrations via the Import Service.  Second, there is a slight limitation to how our Link Tracking Service works.  As you will see below, everything works in the end, but the user experience suffers a little until you finalize your links.

Linking options

Related your choice of connection options is the choice of Link Tracking Service options.  The Quest Link Tracking Service is an optional feature that keeps track of all the Notes documents you have migrated and dynamically redirects users to the current location.  I won’t go into all the details of the service here, but I want to focus on how the Link Redirector page works. 

If you enable the Link Tracking Service, every Notes DocLink (or HTTP link to a web enabled Notes document) in every migrated document gets converted to an HTTP link to a Link Redirector page (QuestLinkTracking.aspx).  This redirector page typically performs a lookup in a centralized Link Tracking database and then dynamically redirects the user to another migrated document in SharePoint (if it has been migrated) or to the original Notes version (if it has not yet been migrated).  So the natural question here is: Where does this Link Redirector page live and how does it get installed?

There are actually now two different versions of the Link Redirector page that you can choose from.  First is the “classic” one that you get when you install the Front End Services solution described above.  This one is configured on a per site collection basis, alongside the Quest Import Service.

An alternative version is the Sandbox Link Redirector page.  This version is intended for cases where you do not have the ability to install custom solutions and/or you cannot establish SQL connections from your server to the a shared Link Tracking database.  The main case we were thinking of when we designed this solution is Microsoft’s Office 365 environment and other highly secured hosting environments, but there will be plenty of people who prefer this option even for on-premises environments.  This page is packaged as a simple SharePoint solution (Quest.SandboxLinkRedirector.wsp).  Because it is a sandbox safe solution, it can actually be installed by any site collection administrator, even on locked down environments such as Office 365, without involving your farm administrators at all.

image

Note that because Sandbox Safe Link Redirector page does not connect to an external Link Tracking database, it always offers to redirect user to Notes, even if the document had been migrated to SharePoint.  In this scenario, users will not actually get redirected to their new SharePoint documents until their links are Finalized.  The saving grace here is that you can Finalize your links as often as you want to.  In productions migrations, customers often choose to Finalize links on a daily basis.

image

Putting it all together

Wow that is a lot of options and choices here!  Let me try to simplify things with a nice table.

Migration mode Quest Import Service SharePoint 2010 Web Services
SharePoint versions 2007, 2010 2010
Office 365 (BPOS) Dedicated only Dedicated and Standard
Server installation Administrator must run MSI, etc. None
Server configuration Per site collection None
Performance Fastest Slowest
Functional limitations None None
Link Tracking Service Full Dynamic Link Redirection (via Front End Services solution) Limited Redirection (via Sandbox Link Redirector solution) *

* NOTE: Strictly speaking, it is possible to install the Front End Services solution (with full dynamic link redirection using a Link Tracking database) even if you are not installing the Quest Import Service.  We believe, however, that most people will either want to install the full solution or keep things as light as possible and will not often mix and match.

Migration basics: Migrating to Document Libraries

From the perspective of a migration specialist, a library is very similar to a list. The main difference is that in a library, each “document” is an actual binary file with various data properties associated with it.

Therefore, migrating a Notes database to a SharePoint document library could be as simple as extracting binary file attachments out of each Notes document and placing them in the library.  This simplistic approach makes sense if the Notes application itself was designed to manage binary files—that is, if each Notes document is really just a wrapper around a binary file attachment.  Domino.Doc is an example of this type of application.  In the screen shot below Notes Migrator for SharePoint was used to extracted the attachments from each Notes document and placed them in a document library.  You can also extract various metadata items about each document and map them to SharePoint properties.

image

Be aware that several things can go wrong with this type of migration job. If there are no attachments in a particular Notes document (i.e., if it is just a normal rich text document), then nothing will be migrated to the library.  If there are multiple attachments in a particular Notes document, they may all be migrated to the library but they will no longer be one self-contained document.  In either case, you have probably misinterpreted the way the Notes application was used, and it should not have been migrated to a library in this manner. 

There are two answers to this dilemma.  The first possibility is to migrate the application to a list instead if a document library.  Even if it was called a “document library” in Notes, it may be more appropriate to map it to a custom list in SharePoint.

The second possibility is generate new documents (one for every Notes document) and check those into the document library.  Notes Migrator for SharePoint can convert Notes documents to a variety of file formats, including HTML, MIME, Word, PDF, and InfoPath.  These are the binary files that you check into user will open when they click on “the document”.   The three most popular choices for formatting Notes documents as binary files are discussed below.

Microsoft Word

Microsoft Word is a popular choice in environments that have standardized on Microsoft Office for document creation. The integration between Microsoft Office and SharePoint libraries is very good and can enable you to build a variety of powerful applications. Users can open documents from libraries, edit them and seamlessly save them back again. If a version control, check-in/check-out, or approval process or workflow has been enabled for the library, it will all work automatically. Office clients even support single sign-on with SharePoint and SharePoint Online Dedicated.

Office documents in SharePoint libraries are easy to search and you can even generate an instant SharePoint workspace to enable teams to collaborate on a particular document. Multiple users can open the same Word document and edit it at the same time, and users can see the changes being made by other users almost instantly.

When migrating Notes documents to the Microsoft Word format, you can migrate to simple unadorned Word documents or to custom Word templates. You can also migrate Notes data items to Word document properties or even to content controls in your custom template.

The following screen shot shows a Notes rich text document that was converted to a Word document on a custom letterhead template and checked into a SharePoint document library:

image

For more details on migrating to Word documents, see these posts.

PDF

PDF is another popular choice for migrating rich text Notes documents. Many organizations, especially in Europe, use PDFs to archive old content. Since PDF is now an open standard, the assumption is that there will always be a PDF reader such as Adobe Acrobat available in the future. An organization that has a large number of Notes databases with rich text documents may find that PDF is an ideal target format for many of them.

When PDF documents are placed in a SharePoint library, the integration is not quite as tight as it is with Office applications, but the user experience is still reasonable. Even though PDF readers and editors are not generally “SharePoint aware,” the experience of opening PDF documents is similar to downloading them from any web site. PDF documents can work with SharePoint’s search features, but you need to install a free add-on from Adobe for SharePoint’s full-text search indexer to read the content.

A word of warning about migration tools: a number of tools advertise the ability to convert Notes documents to PDF documents, but deliver poor results. If you plan to use this feature, we strongly recommend that you test the tools with your user’s most complex documents. Watch for how nested tables, embedded images, links to attachments and doc links are handled.

The following figure shows a Notes rich text document that was converted to a PDF document and checked into a SharePoint document library:

image

PDF migrations are discussed in more detail here.

Document rendering

In some cases, simply converting the rich text bodies of your Notes documents to Word or PDF files is not good enough because it does not capture the rich form layout that Notes users are used to. Without the form layout, you really haven’t captured all the information.

This is where an advanced concept known as document rendering comes in. With this technique, the migration tool “renders” each document with its original Notes form to generate a new rich text document that includes the entire form layout you had in Notes. To visualize this, compare the generated PDF document in the following figure with the PDF example provided above:

image

In addition to the rich text body, we captured the information that was presented with the original Notes form. Most significantly, we did not have to redevelop the form in SharePoint to accomplish this, nor did we have to explicitly map all the data fields that are displayed in the form header.

We used PDF in this example, but the same concept of rendering documents with forms applies equally well to Word documents, InfoPath documents, pages and even lists.

The Render With Form feature is discussed more fully here.

InfoPath

People sometimes choose the InfoPath document format when they want to migrate complex Notes applications to SharePoint, usually for one of two reasons: either the applications have complex data structures that do not lend themselves to being stored in a SharePoint List, or the applications have complex form designs that contain dynamically hidden sections, input validation rules, buttons, form events, or other sophisticated form logic. Ways of addressing the second issue are discussed in the section “Migrating Application Designs” below, so for now, we will focus on the migration of Notes content to InfoPath documents.

InfoPath data documents (traditionally called “InfoPath forms”) are really XML files that you edit using the InfoPath client (part of Office) or perhaps in a browser if your SharePoint server is running InfoPath Form Services. You specify the layout and behavior of your InfoPath form (and associate it with your desired XML schema) by creating an InfoPath form template. Typically you would store your XML data documents in a special type of SharePoint library known as a form library. This library is associated with one or more form templates in such a way that end users get a fairly seamless experience of creating, viewing and editing complex documents right from the library.

When performing a Notes migration, your main job is to convert Notes documents to InfoPath XML data documents according to a particular XML schema and check them into a form library.  Notes Migrator for SharePoint makes this fairly straightforward.  All you need to do is load in an InfoPath form template and specify how you want various Notes data elements to map to the various parts of your new XML schema.  These elements might include not only simple data fields, but also rich text, embedded images and attachments, links to external attachments, and links to other documents.  As XML schemas are not necessarily one dimensional, you may need to map one-to-many data items as well (for example, a product description document might contain multiple distributor names).

The following screen shot shows a Notes rich text document that was converted to an InfoPath document (associated with a specific InfoPath form template) and checked into a SharePoint form library:

image

NOTE:  We did not specify how the InfoPath form template was created. It may have been created from scratch by your InfoPath developer or you may have migrated an existing Notes form using Notes Migrator for SharePoint.

For more details on InfoPath migrations, look here.

Details, details, details

Additional considerations for migrating to SharePoint libraries include:

  • Notes documents contain metadata such as Created By, Created Date, Last Modified By, and Last Modified Date. Many migration tools drop this metadata during migrations to SharePoint, resulting in a major loss of business data.
  • Most Notes databases contain access control lists, which determine what specific users can do in a particular application. In addition, individual documents contain access restrictions such as Readers lists and Authors lists. Access definitions may use groups in the Domino Directory as well as roles defined for the database. Preserving all this information correctly in SharePoint may be critical to a successful migration of sensitive data.
  • If your Notes application has a concept of document versioning, make sure your migration tool allows you to correctly map the versioning to SharePoint versioning. All versions of a given document in Notes should appear to be versions of the same document in SharePoint.
  • It is common to want to assign folders during a migration. You can dynamically generate folder names in SharePoint based on data extracted from Notes.
  • Instead of folders, you may want to use a powerful new SharePoint feature called Document Sets. This feature is discussed in more detail here
  • When generating documents in a document library, you may with to put images and attachments in a subfolder or even in a different library.  This technique is discussed here.
  • For complex applications with mixed content, may wish to assign Content Types to documents as you migrate them.
  • Lists, Libraries and Pages

    There are three basic ways to store content in SharePoint: lists, libraries, and pages. Each of these has a number of interesting variations, but it is important to understand the differences between these three fundamental types so you can best decide what you want to migrate to. Each type is described briefly here; subsequent posts will explain in detail how to migrate content to each from Lotus Notes applications.

    Lists

    Lists are similar to tables in a relational database. A list is a flat collection of data records (called items in SharePoint) with a fixed set of data fields (called columns). Each data column has a fixed name and type. For example, a customer list may have a Text column called “Customer Name,” a Date column called “First Purchase Date,” and potentially dozens of other columns. One particularly interesting column type is Rich Text (also known as a “Note” “Body” column); this is where one would typically store large amounts of rich text. Lists can also have one or more binary attachments and may have one or more views, which allow users to select and sort the items in various ways.

    All of this should sound pretty familiar to Notes customers, because a list is actually the closest thing in SharePoint to a Notes database. The biggest difference is that SharePoint lists are highly structured with a fixed schema (like a relational database), whereas Notes databases can be very unstructured, with every document having a different set of data items.

    Libraries

    Libraries are collections of binary files, such as images, Word documents, or audio clips. While lists and libraries are very similar internally, the metaphor is very different: in a list, the document may contain several binary file attachments; in a library, the binary file is the document. The emphasis in libraries is the document management functionality, including versioning and check-in/check-out. As with lists, libraries can have many additional data columns defined for capturing additional information about each document.

    In the Notes world, the closest thing to a SharePoint library is a Domino.Doc file cabinet. (Domino.Doc was a popular document management system built on top of Notes.) Many organizations also built custom Notes applications that attempt to implement document management functionality. Any time you see a Notes application where the file attachment is “the document,” consider migrating it to a SharePoint library. It is also common for Notes “team site” applications to have a document library section as part of the overall application.

    Pages

    Pages are the building blocks of all SharePoint sites. These are the web pages you actually see in the web browser every time you click on a link to view a site, open a document, enter some information, or do just about anything else. Most people do not realize that the same pages that make up the sites themselves can also be used as data documents. SharePoint actually allows you to create several types of content pages, including basic pages, wiki pages, web part pages, and publishing pages. SharePoint Online Dedicated includes several nice publishing site templates that are designed to manage the authoring, approval and display of rich text web pages.

    While content pages have no exact equivalent in the Notes world, they can be a great way to migrate certain types of Notes applications. Any time you see a Notes application in which the main intent was to publish a library of rich text pages to a large number of users, consider migrating it to a SharePoint page library or a publishing site. This includes the many Notes applications that implemented public web or extranet sites.

    Office 365 launches… Now read the white paper!

    Last week, Microsoft  announced the launch of Office 365

    One week before that, my awesome new white paper, “Migrating Lotus Notes Applications to Microsoft Office 365 and SharePoint Online”, went live on the Quest web site.

    Here is a sneak peek at the table of contents:

    Introduction
    Pre-Migration Analysis
        Introduction
        Discovering All of Your Notes Databases
        Determining What You Don’t Have to Migrate
        Understanding the Complexity of Your Applications
        Consolidating Similar Application Designs
    Formulating your Migration Plan
    Migrating Notes Application Content
        Application Migration Overview
        Lists, Libraries and Pages
        Migrating Content to Standard Lists
        Migrating Content to Custom Lists
        Migrating to Document Libraries
        Generating Documents
        Using Document Sets
        Migrating Content to SharePoint Pages
        Archiving Legacy Content: Document Rendering
    Migrating Application Designs
        Migrating Schema from Notes Applications
        Migrating Form Designs to InfoPath
        Migrating Approval Process and Workflow State
        Deploying Sandbox Solutions
    Understanding the Limitations of Office 365
        Connecting to Enterprise Data
        Mail-In Databases
        Deploying Custom Code
        Standard Server Settings
        Bandwidth and Throttling
        Other “Missing” Features
    Conclusion
    Appendix A – SharePoint Online Deployment Basics
        Getting Started
        Choosing the Right Office 365 Plan
        Basic Tenant Administration
        Adding Users and Synchronizing Directories
        Provisioning and Managing Site Collections
        Building your SharePoint Sites

     

    Let me know what you think!

    PS:  A similar paper on migrating to SharePoint Online Dedicated (aka BPOS-D) is going through post-production now.  Stay tuned.

    Utilizing Content Types during a Notes to SharePoint migration (Part 2)

    Part 1 of this article [link] describes how SharePoint Content Types might be useful when migrating multiple Notes “document types” to a single SharePoint List, Document Library, or InfoPath Forms Library.  Now we will take a look at how this can be accomplished using our tool, Notes Migrator for SharePoint.

    First a recap of the Content Type features in the Notes Migrator for SharePoint documentation:

    • To assign the Content Type of a SharePoint item, simply add a Text field named “ContentType” and map the appropriate input data to it.
    • The ContentTypes property in a Source DataDefinition field indicates that the data field should only be migrated SharePoint for certain Content Types.  Leave this property blank to always write the field, regardless of content type.

    The following example walks through the steps needed to create a SharePoint list using Content Types and then migrate Notes content to it:

    Our example Notes database is a fairly typical Notes document library except that it has three types of documents in it:  Tech Notes, Code Samples, and FAQ Documents.  The “document type” is really controlled via a DocType field (not separate forms as described in Part 1).  All documents contain standard fields such as “Subject”, “Category”, and “Product”.  Code Sample documents contain an extra “Language” field and FAQ Documents contain “FAQ Type” fields.

    image

    On our SharePoint site we created a corresponding List with three content types similar to the ones we had in Notes:

    • Content type “Tech Note” has columns “Title”, “Product”, “Details”, “Url” (a “link” field).
    • Content type “Sample” inherits from “Tech Note” and adds column “Language” (a choice field).
    • Content type “FAQ” inherits from “Tech Note” and adds column “FAQType” (a choice field).

    Note: I intentionally designed the Notes “FAQType” field to have a slightly set of choices than the WSS “FAQType” field so we can demo data massaging.

    In the following screen shots, notice that certain columns only apply to certain Content Types:

    image  image

     

    For the migration job itself, I selected all the data fields, including the DocType field as well as fields that were specific to certain document types, in my Source Data Definition.  Nothing unusual here, except that I did a little “data massaging” along the way:

    • The “Product” column is really the Categories item with an alias.
    • The “FAQType” column is a Formula column that massages the data during migration: 
      @If(FAQType=”How To” | FAQType=”Problem”;”Solutions”;FAQType)
    • The “OnlineLink” column is a Formula column that computes the URL to the document on the Proposion web site.

    image

    In our Target Data Definition, we added a SharePoint field called “ContentType”.  We can map any source column we want to this field, and the Content Type for each new document will be set accordingly.  In our case, of course, we will map the DocType column to this field.

    image

    For the “Language” and “FAQType” fields, which are only for certain Content Types, we specified the content type in the ContentTypes property of the Target Data Definition field definition.

    image 

    The result of running this migration job is a SharePoint List populated with mixed types of content:

    image

    Utilizing Content Types during a Notes to SharePoint migration – Part 1

    Content Types are a very powerful way to express the concept of “document types” or “business objects” in SharePoint. 

    In the Notes world, it is quite common to design an application that has multiple types of documents in it.  Even though Notes databases are unstructured and (by design) lack any type of schema, good Notes developers usually adopt certain patterns and practices that impose some order in their applications and usually this includes establishing distinct, meaningful document types.

    From a development perspective, this is largely driven by the different Forms used to create Notes documents.  A great example of this is a standard Notes mail database, which has Memos, Tasks and Calendar Entries.  If you want a business application example, think of Customers, Customer Contacts, and Sales Reps in a CRM database.  Each of these “document types” has different fields in it and each is created with a different Form. 

    image

    While Notes Forms has several other uses (for example, to display a pop-up dialog), the Form used to a create a document becomes the de facto “document type” in most cases.  An application may contain one or more Views or Folders for each document type (for example “All Tasks by Priority” or “Open Tasks”).  But it is also common to design a View or Folder that shows multiple document types together (for example showing Customers and Customer Contacts together in the same view).

    A common variation on the above pattern is to use different forms to express extended document types (what object oriented developers would call “sub-classes”).   An example of this would be an application that had both Customers and Government Customers.  While a Government Customer is a type of Customer, it has additional fields and may have additional data validation rules, different security, etc.  You could easily imagine a View that shows “all customers” together without making any distinction, but when you opened a Government Customer record you would see something different than if you opened a “plain” Customer record.

    The notion of defining different document types is not limited to Notes applications.  QuickPlace sites have several different default “Page Types” and users can create their own custom Page Types with additional fields.  Similarly Domino.Doc libraries can be extended with both custom Document Types and custom Binder Types.

    So how do these concepts translate to the SharePoint world and how do you you deal with them when migrating?

    When migrating a Notes application, a QuickPlace site or a Domino.Doc library to SharePoint, developers typically do one of three things.  Note that each of the following choices apply equally well whether you are migrating to a SharePoint List, Document Library, or an InfoPath Forms Library:

    1. Split different document types into a different Lists.  You could easily run one migration job that moved all Customer records to one list and another job that moved all Sales Rep records to a different List in the same SharePoint site.  This would be a particularly good idea if, for example, you wanted Customers to have a “Rep” field that was a Lookup field from the Sales Rep list.  It is less clear, however, that it would be good to split up Customers and Government Customers this way as you probably want them to stay together.  Notes Migrator for SharePoint allows you to select records by View or by Formula; if you can not find a View that selects the documents you want you can use a Formula such as: FORM = ‘GovernmentContact’
    2. Merge different document types into a single uniform List schema.  For example, you could migrate all Customers to a single Customers list and discard the extra fields that was in Government Customers (a “least common denominator” approach).  Or you could include all the “government” fields in your new Customer definition and just leave these blank for the records they don’t apply to (a “grab everything” approach).  You might even add a new “Customer Type” field that recorded whether or not it was a Government Customer in Notes.
    3. Use Content Types.  The pattern of using Forms to express document types in Notes applications maps almost perfectly to SharePoint “Content Types”.

    image

    There is a lot of information on the web about Content Types.  I like Andrew May’s blog [link].  The Microsoft SDK documentation [link], which can be pretty intimidating to read, introduces Content Types this way:

    Content types, a core concept used throughout the functionality and services offered in Windows SharePoint Services 3.0, are designed to help users organize their SharePoint content in a more meaningful way. A content type is a reusable collection of settings you want to apply to a certain category of content. Content types enable you to manage the metadata and behaviors of a document or item type in a centralized, reusable way.

    One of the reasons Content Types seem so intimidating to new SharePoint developers is that they are used for so many things.  Not only are they used to express which document types contain which fields (the important part for people migrating content), they can also be used for the following:

    • Custom user interfaces
    • Templates for creating new documents
    • Workflow rules
    • Microsoft Office integration
    • Inheritance from “base” Content Types
    • Reuse across multiple lists in a SharePoint site

    The good news is that using Content Types for simple things (i.e., defining multiple structured document types in a single list) is actually pretty simple.  In spite of what some of the literature implies, you do not need to be a developer or create features or even use XML to create document types.  I believe that as people come to understand them better, they will become more widely used.

    image

    Notes Migrator for SharePoint has the ability to set content types in migrated documents based on the Form that was used in Notes, or whatever other criteria you want to use.  Part 2 of this article will discuss how we make that happen and will include a working example.

    Getting Started with Migrating to SQL Server Data Normalization: Images and Attachments

    In yesterday’s article [link] I provided a walkthrough of migrating to a flat SQL Server table.   Now we will start digging into data normalization with related tabled.  Here we will look at using a One-To-Many table to store binary objects and in the next article we will examine Lookup tables.

    Imagine that I designed a SQL Server database with a main “Documents” table and related “Parts” table for storing migrated images, attachments and embedded objects.  I wanted a separate table here because there is a one-to-many relationship between documents and images. 

    The basic design for my database is shown below.  In addition to the BinaryData column (type VarBinary) to hold the raw binary data, I added DocID (int) and PartID (string) columns which together constitute a unique primary key.  Finally there is a PartName column (string) which stores the friendly display name for the attachments.   

    image

    Now in my Target Data Definition, I specify my Parts table as a “Normalized Table” with the relationship One-To-Many.  If you are lost at this point, go back and review the prior article “Getting Started with Migrating to SQL Server Databases”. 

    image

    In the Data Fields tab of my Target Data Definition, I need to define two columns for migrating content to.  (Unfortunately the “Load From SQL Server Database” button is not smart enough to do a complete job here, so you will have to do much of this manually.  If you press that button, make sure you review the following details carefully afterwards.) 

    First I defined my DocId field for the Parts table and I set the “Link To Primary” property to True.  This tells Notes Migrator for SharePoint that the Primary Key of the Primary Table should always be copied to the DocId column of the Parts table.  This will happen regardless of whether the Primary Key was set from Notes data or was an auto-generated Identity column.

    image

    Next I defined a field of type File.  I called it “BinaryData” but I could have called it anything I wanted to.  Using the File Data Column, File Key Column, and File Name properties for my File field, I specified the three columns in my database described above for storing information about the migrated binary files.  The File Data Column property indicates where the actual binary data should go (the BinaryData column in my case).  The File Key Column property indicates where the unique attachment/image/object ID should go (PartID).  The File Name Column property indicates where the friendly display name should go (PartName).  Note that the File Name Column property is optional and the values written to the indicated column are not guaranteed to be unique.

    image

    Finally I set the Is Gating Field property for my File field to True.  This is very important, as it controls the normalization of records; if I map an array of 10 values to this field it will cause 10 records to be written to the Parts table. 

    As shown below, I actually map all the attachments, images, and embedded objects to this field.  So if a Notes document had 6 attachments, 2 images and 3 embedded objects, we would get 11 new records in the Parts table.

    image

    After running my migration job, we can see that the DocID column values in the Documents table are automatically correlated with the DocID column values in the Parts table, and that it is not a one-to-one relationship.

    image

    One final question remains here… If all my images and attachments are stored in a separate table, how does the migrated rich text field (which is now an HTML field in the Documents table) refer to them?  I controlled this by using the “Generating Links to SQL Records” options on the Advanced tab of the migration job.  Here you can specify how links to images, attachments and even other documents should be formatted in the migrated rich text.  This feature is optional, and only makes sense if you have designed a web application to display the images and attachment files in response to a web request. 

    image

    Using the option shown above, the following attachment link and image reference tags were generated in my rich text document:

    <a href=”http://quest.com/parts?server=REMGW0MTL1%5cSHAREPOINT&database=QDemo&table=Parts&id=2382&part=AttachmentLinks.docx”>

    <img src=”http://quest.com/parts?server=REMGW0MTL1%5cSHAREPOINT&database=QDemo&table=Parts&id=2382&part=image0.GIF” border=”0″ />

    Getting Started with Migrating to SQL Server Databases

    Notes Migrator for SharePoint 6.0 now allows you to migrate Lotus Notes content directly to SQL Server databases.  We have seen three main cases where customers are considering doing that as opposed to migrating to SharePoint lists and libraries:

    1. They do not feel that SharePoint Lists are adequate to the needs of the job.  In this category are concerns about SharePoint scalability and volume, as well as the lack of support for transactions.
    2. They want to access the data outside of SharePoint for some reason.  Perhaps the data needs to be accessed by multiple applications (CRM systems, etc.) or the enterprise is trying to achieve a consistent architecture for modeling/storing/archiving all types of business data.
    3. They feel that they can rebuild the application design more effectively using .NET, Java or some other app dev tool.

    Some people consider #1 and #3 to be issues of “application complexity”:  If an app is too complex, it should become a SQL Server / .NET solution because they assume that SharePoint can’t handle it.  While we never completely agreed with this thinking, and with SharePoint 2010 it is even less true, there are still good reasons to want to migrate to SQL Server and we are happy to start supporting it.

    This article covers migrating to a simple flat SQL Server table.  Migrating to multiple tables (and normalizing the data along the way) is a more complex subject that will be covered in a subsequent article.  For now, know that this feature is capable of maintaining separate tables for binary images and attachments, normalizing multi-valued item to related tables, and managing references to lookup tables.

    Migrating to SQL Server is currently only supported in the Designer client, not the Migration Console.  The first thing you need to do is start Designer, go onto the tool Options, and enable SQL Server migrations in the last tab.  You should also specify the credentials you want to use for connection to SQL Server here.

    image

    Once you have done this, you will now see a new drop-down selector for the target type on the target data definition tab.  This is where you indicate that a particular job is intended for SQL Server instead of SharePoint.

    image

    Press the Select button to specify the database you want to connect to.  (Sorry there is no “picker” here yet but there will be.)

    image

    Back on the SQL Server tab in your migration job, you can verify your connection using the Test button.  Remember that you can change your credentials for connecting to SQL Server in the Options dialog, as described earlier.

    image

    Next press the Create button to start designing a target data definition.  As with SharePoint target data definitions, this is where you define the schema for the target you are going to migrate to.  (One big difference here is that our tool does not currently allow you to provision SQL Server tables – you have to define your new schema using SQL Server Management Studio or your favorite design tool.) 

    The first tab of the SQL Server data definition editor is where you specify the table you want to migrate to.  Specify the Primary Table using the drop-down picker.  (We will explore the use of additional “Normalized Tables” in a subsequent article.  For now, lets stick to just one table.) 

    image

    On the second tab is where you define all the fields in your target table.  You can specify them manually if you want to, but the easiest way to do this is usually to just press the “Load from SQL Server database” button.  This should populate the data definition with all the fields you might want to migrated to.

    image

    Each target field has a number of properties that control how Notes Migrator for SharePoint will migrate data to it.  The most important of these is the Type property, which may be set to a variety of standard SQL Server column types.  We also have types for documents that our tool knows how to generate (Word, PDF, and InfoPath documents) which may be written to SQL Server as binary files.

    image

    A few other are other properties worth mentioning here: “Is Html Field” controls whether or not the tool should perform the usual rich text fix-up for DocLinks, attachment references, etc.  “Is Windows Name” controls whether the tool should attempt to map Notes names to Windows names using the configured user/group mapping options.

    image

    Two interesting properties relate to columns that are part of the table’s primary key.  These are not critical in a simple flat table like this, but become very important when you get into data normalization (more later). 

    image

    Save your data definition and then proceed to the Map Data tab on your migration job.  The mapping of Notes columns to SQL Server field is the same as it is for SharePoint fields, including the ability to use formulas or constant values.  

    Note that if a column is an auto-numbering Identity Column (like the DocID column shown above), you have a choice as to whether you map Notes data (such at the Notes ID) to this column or let SQL server assign the values for you.  If you want the later, simply refrain from mapping anything to this column.

    image

    One thing that is missing on the Map Data tab is the checkboxes for auto-mapping document metadata, etc.  Similarly on the Advanced tab there is no option for mapping permissions.  The reason is that SQL Server does not have explicit support for these types of constructs the way SharePoint does.  Of course, you may choose to add columns to you table that capture such information and build applications that use them.  In the above example, we are mapping the Created Date and Created By metadata to SharePoint columns, but from the tool’s perspective there is nothing really special about them.

    When you are satisfied with your mappings, press the Run Job button and start migrating!  You can use tools such as SQL Server Management Studio or Quest’s own Toad for SQL Server to view the results.

    image

    One issue that comes up when migrating to SQL Server is that the server tends to be very sensitive regarding data validation.  Whereas the SharePoint APIs tend to be more forgiving and clean up data as you go, SQL Server will throw a fatal exception and an error message.  While we hope to get a little smarter about this over time, Notes Migrator for SharePoint currently gives up and simply displays the SQL Server error messages in many cases.  The result for you may be that a little trial and error needed in order to achieve a successful migration.

    Note that, unlike with SharePoint, our tool does not give you the option to delete the prior contents.  You are always in “append” mode here so if you want to re-migrate you will need to clear the results of your previous job manually.  There are other things we did with SharePoint migrations that we could not do with SQL Server migrations:  detecting and merging duplicates, out-of-band metadata migration, and out-of-band security mapping. 

    The good news here is that in the SQL Server world, we finally have transactions.  So if a large migration fails halfway through, we can (and do) roll back the entire job.

    One final observation about the brave new world of migrating to SQL Server:  SharePoint 2010 now makes it easier than ever to bring SQL Server data back into the SharePoint user interface using External Lists, InfoPath List forms, etc., so you can think of this new feature as giving you a choice as to where the back-end data lives, even when you know you are going to build the front-end in SharePoint.