Lotus Notes to SharePoint Blog

Blog about Dell's Notes Migrator to SharePoint tool and other things related to Lotus Notes migration projects

Category Archives: Word

Automatically ZIP file attachments

Many customers have requested the ability to compress Notes file attachments while migrating them to SharePoint. There are a number of good reasons for wanting to do this:

  • Save disk space on SharePoint server
  • Get around SharePoint file restrictions (i.e., blocked file extensions and/or size limits)
  • Improve the bandwidth sending data to remote SharePoint servers
  • Eliminate problems (hangs and memory leaks) when embedding certain types of file attachments inside Word documents

Notes Migrator for SharePoint 6.2 now allows you to achieve this in Notes and Domino.Doc migration jobs via a new property on Attachment columns in your Source Data Definition. The “Compress” property may be set to “None” (the default) or to “Zip”.

clip_image002

You can also configure a set of global exceptions to this rule on the Notes tab of the tool’s Options dialog. The “Compression Exclusions” option allows you to specify any file extensions that should never be zipped. This would typically include media files that are already well-compressed and would not benefit from zipping.

clip_image004

When the Compression property is set to “Zip”, all extracted attachments will be compressed and placed inside a ZIP file when migrated. As shown below, any icons or text links to the attachments will appear the same as before, but when the user clicks on them they will open up a ZIP file instead of the “raw” file.

clip_image006

Additional Notes

Zip files will always be excluded from further zipping, even if they are not specified in the “Compression Exclusions” list.

If you wish to use this feature to migrate files that were previously blocked by SharePoint (for example .EXE files), be sure to remove these extensions from your Blocked Files list on the SharePoint tab of the tool’s Options dialog.

When using this feature with Domino.Doc sources, add a custom Attachment column as described above. Map this custom column (instead of the predefined {Attachments} column) when mapping attachments to a target column.

clip_image008

This feature is not currently available to QuickR or QuickPlace.

Attachments in generated Word documents… use caution!

We have had a number of support cases on this over the last year, and we have made an effort to clarify our position on this in our latest release notes:

Migrating file attachments inside of MS Word documents is not recommended for large migration jobs. Since Word attachments are implemented as “Packager” OLE objects, the migration process is forced to invoke Packager code as well as the OLE handlers for the file type in question (often in separate processes). The problem is that each type of attachment is handled differently depending on which type of application created the attachment. So (for example) the first 1000 attachments may work fine and then document 1001 has a different type of attachment that causes a memory leak when Microsoft converts it into a Packager object. When migrating attachments and OLE Objects to embedded objects in MS Word, it is highly recommended that the workstation performing the migration has native applications installed that can open and edit every type of attachment that the migration jobs will be encountering during the migration.

The two recommended workarounds are:

  • Migrate attachments separately to the SharePoint document library. (The links from the Word documents to the attachments will be preserved.)

  • Place all attachments inside ZIP files inside the Word documents. (This is a new feature for version 6.2.)

If you still insist on embedding attachments into Word documents, you should understand that you are doing so at your own risk and that Quest support will probably not be able to help you if problems occur. Here are some tips that have helped other customers in the past:

  • There are known issues with packaging Adobe Acrobat 10 objects. If you have PDF attachments, be sure to install Adobe Acrobat Reader version 9 on the workstation.

  • Packager leaks usually have to do with cases where packager cannot use type-specific COM objects (including all .MSG files, and including .PDF files when the PDF reader is not installed)

  • Leaked resource handles accumulate per Windows process. This means that you should restart NMSP Designer between large jobs, even if they are successful. It also means that batches of jobs in the NMSP Migration Console would still not be a good idea.

  • There may other issues with packaging certain types of objects. Since we cannot product how third-party OLE handler will operate, this is beyond our control.

New Webcast: Migrating Lotus Notes Applications to SharePoint Online in Office 365

Migration basics: Migrating to Document Libraries

From the perspective of a migration specialist, a library is very similar to a list. The main difference is that in a library, each “document” is an actual binary file with various data properties associated with it.

Therefore, migrating a Notes database to a SharePoint document library could be as simple as extracting binary file attachments out of each Notes document and placing them in the library.  This simplistic approach makes sense if the Notes application itself was designed to manage binary files—that is, if each Notes document is really just a wrapper around a binary file attachment.  Domino.Doc is an example of this type of application.  In the screen shot below Notes Migrator for SharePoint was used to extracted the attachments from each Notes document and placed them in a document library.  You can also extract various metadata items about each document and map them to SharePoint properties.

image

Be aware that several things can go wrong with this type of migration job. If there are no attachments in a particular Notes document (i.e., if it is just a normal rich text document), then nothing will be migrated to the library.  If there are multiple attachments in a particular Notes document, they may all be migrated to the library but they will no longer be one self-contained document.  In either case, you have probably misinterpreted the way the Notes application was used, and it should not have been migrated to a library in this manner. 

There are two answers to this dilemma.  The first possibility is to migrate the application to a list instead if a document library.  Even if it was called a “document library” in Notes, it may be more appropriate to map it to a custom list in SharePoint.

The second possibility is generate new documents (one for every Notes document) and check those into the document library.  Notes Migrator for SharePoint can convert Notes documents to a variety of file formats, including HTML, MIME, Word, PDF, and InfoPath.  These are the binary files that you check into user will open when they click on “the document”.   The three most popular choices for formatting Notes documents as binary files are discussed below.

Microsoft Word

Microsoft Word is a popular choice in environments that have standardized on Microsoft Office for document creation. The integration between Microsoft Office and SharePoint libraries is very good and can enable you to build a variety of powerful applications. Users can open documents from libraries, edit them and seamlessly save them back again. If a version control, check-in/check-out, or approval process or workflow has been enabled for the library, it will all work automatically. Office clients even support single sign-on with SharePoint and SharePoint Online Dedicated.

Office documents in SharePoint libraries are easy to search and you can even generate an instant SharePoint workspace to enable teams to collaborate on a particular document. Multiple users can open the same Word document and edit it at the same time, and users can see the changes being made by other users almost instantly.

When migrating Notes documents to the Microsoft Word format, you can migrate to simple unadorned Word documents or to custom Word templates. You can also migrate Notes data items to Word document properties or even to content controls in your custom template.

The following screen shot shows a Notes rich text document that was converted to a Word document on a custom letterhead template and checked into a SharePoint document library:

image

For more details on migrating to Word documents, see these posts.

PDF

PDF is another popular choice for migrating rich text Notes documents. Many organizations, especially in Europe, use PDFs to archive old content. Since PDF is now an open standard, the assumption is that there will always be a PDF reader such as Adobe Acrobat available in the future. An organization that has a large number of Notes databases with rich text documents may find that PDF is an ideal target format for many of them.

When PDF documents are placed in a SharePoint library, the integration is not quite as tight as it is with Office applications, but the user experience is still reasonable. Even though PDF readers and editors are not generally “SharePoint aware,” the experience of opening PDF documents is similar to downloading them from any web site. PDF documents can work with SharePoint’s search features, but you need to install a free add-on from Adobe for SharePoint’s full-text search indexer to read the content.

A word of warning about migration tools: a number of tools advertise the ability to convert Notes documents to PDF documents, but deliver poor results. If you plan to use this feature, we strongly recommend that you test the tools with your user’s most complex documents. Watch for how nested tables, embedded images, links to attachments and doc links are handled.

The following figure shows a Notes rich text document that was converted to a PDF document and checked into a SharePoint document library:

image

PDF migrations are discussed in more detail here.

Document rendering

In some cases, simply converting the rich text bodies of your Notes documents to Word or PDF files is not good enough because it does not capture the rich form layout that Notes users are used to. Without the form layout, you really haven’t captured all the information.

This is where an advanced concept known as document rendering comes in. With this technique, the migration tool “renders” each document with its original Notes form to generate a new rich text document that includes the entire form layout you had in Notes. To visualize this, compare the generated PDF document in the following figure with the PDF example provided above:

image

In addition to the rich text body, we captured the information that was presented with the original Notes form. Most significantly, we did not have to redevelop the form in SharePoint to accomplish this, nor did we have to explicitly map all the data fields that are displayed in the form header.

We used PDF in this example, but the same concept of rendering documents with forms applies equally well to Word documents, InfoPath documents, pages and even lists.

The Render With Form feature is discussed more fully here.

InfoPath

People sometimes choose the InfoPath document format when they want to migrate complex Notes applications to SharePoint, usually for one of two reasons: either the applications have complex data structures that do not lend themselves to being stored in a SharePoint List, or the applications have complex form designs that contain dynamically hidden sections, input validation rules, buttons, form events, or other sophisticated form logic. Ways of addressing the second issue are discussed in the section “Migrating Application Designs” below, so for now, we will focus on the migration of Notes content to InfoPath documents.

InfoPath data documents (traditionally called “InfoPath forms”) are really XML files that you edit using the InfoPath client (part of Office) or perhaps in a browser if your SharePoint server is running InfoPath Form Services. You specify the layout and behavior of your InfoPath form (and associate it with your desired XML schema) by creating an InfoPath form template. Typically you would store your XML data documents in a special type of SharePoint library known as a form library. This library is associated with one or more form templates in such a way that end users get a fairly seamless experience of creating, viewing and editing complex documents right from the library.

When performing a Notes migration, your main job is to convert Notes documents to InfoPath XML data documents according to a particular XML schema and check them into a form library.  Notes Migrator for SharePoint makes this fairly straightforward.  All you need to do is load in an InfoPath form template and specify how you want various Notes data elements to map to the various parts of your new XML schema.  These elements might include not only simple data fields, but also rich text, embedded images and attachments, links to external attachments, and links to other documents.  As XML schemas are not necessarily one dimensional, you may need to map one-to-many data items as well (for example, a product description document might contain multiple distributor names).

The following screen shot shows a Notes rich text document that was converted to an InfoPath document (associated with a specific InfoPath form template) and checked into a SharePoint form library:

image

NOTE:  We did not specify how the InfoPath form template was created. It may have been created from scratch by your InfoPath developer or you may have migrated an existing Notes form using Notes Migrator for SharePoint.

For more details on InfoPath migrations, look here.

Details, details, details

Additional considerations for migrating to SharePoint libraries include:

  • Notes documents contain metadata such as Created By, Created Date, Last Modified By, and Last Modified Date. Many migration tools drop this metadata during migrations to SharePoint, resulting in a major loss of business data.
  • Most Notes databases contain access control lists, which determine what specific users can do in a particular application. In addition, individual documents contain access restrictions such as Readers lists and Authors lists. Access definitions may use groups in the Domino Directory as well as roles defined for the database. Preserving all this information correctly in SharePoint may be critical to a successful migration of sensitive data.
  • If your Notes application has a concept of document versioning, make sure your migration tool allows you to correctly map the versioning to SharePoint versioning. All versions of a given document in Notes should appear to be versions of the same document in SharePoint.
  • It is common to want to assign folders during a migration. You can dynamically generate folder names in SharePoint based on data extracted from Notes.
  • Instead of folders, you may want to use a powerful new SharePoint feature called Document Sets. This feature is discussed in more detail here
  • When generating documents in a document library, you may with to put images and attachments in a subfolder or even in a different library.  This technique is discussed here.
  • For complex applications with mixed content, may wish to assign Content Types to documents as you migrate them.
  • Generating Microsoft Word Documents

    image

    This feature allows you to generate Microsoft Word 2007 / 2010 (OpenXML) documents from Lotus Notes documents. Microsoft Word is a very flexible and powerful environment and NMSP supports a wide range of migration scenarios:

    1. Migrate Notes rich text to new (blank) Word document – In this scenario the user wants to map the rich text portion of their Notes documents (typically the “Body” field) to Word 2007 / 2010 documents (DOCX files) created from scratch. The generated rich text content is pretty much the only thing in the resulting documents. The fidelity of the migrated content is very high and includes tables, fonts, bullets, images, doc links, and much more. The known limitations are listed at the end of this document.

    Optionally, the user may want to migrate specific Notes data items to Standard Document Properties such as Author, Category or Description within the generated Word documents.  As these documents are generated, they will be checked into the designated SharePoint library.

    A large number of existing Notes Migrator for SharePoint features apply here, including the ability to set created/modified metadata, document permissions, target folders, content types, workflow state, version history, data transformations and more.

    Furthermore, the user may want to migrate specific Notes data items to specific metadata columns in the document library. When the document is opened in SharePoint, these properties will appear as “Server Properties” in the Word user interface.

    2. Migrate Notes rich text to new Word documents based on existing templates – This scenario is similar to the one above except that the user can specify the Word 2007 / 2010 template (DOTX or DOTMX file) they want to start with. The generated documents will pick up the headers, footers, backgrounds, etc., of the original template. If the template includes any rich text content, the migrated content will be appended to it.

    If the Word template defines Custom Document Properties, the user may want to migrate specific Notes data items to these Custom Properties as well as to the Standard Properties described above.

    3. Migrate Notes documents to Word documents with embedded content controls and legacy controls – In this scenario, the user is using the Word template as a “form” with rich layout and data entry fields (and possibly data validation rules, computations, actions, etc.). For each Notes document, Notes Migrator for SharePoint will generate an equivalent Word document with specific Notes data items mapped to these controls.
     
    There are several distinct cases here:

    3a. If the user is taking advantage of SharePoint’s automatic mapping of SharePoint meta-data columns to Word content controls (exposed to SharePoint developers as “Quick Parts”) then the best approach is to simply map the Notes data items to the metadata columns in the document library and let SharePoint synchronize them with Word.

    3b. Notes Migrator for SharePoint can also set the contents of the content controls or legacy controls directly with the mapped Notes data. They will continue to appear as data entry fields in the document, but they will not be synchronized with SharePoint metadata columns.

    3c. Notes Migrator for SharePoint can also replace the content controls or legacy controls entirely with the mapped Notes data. They will no longer appear as data entry fields.

    Note that in each of the above three cases, the user may still want to map a Notes rich text “Body” field to the document in addition to populating the controls, as described in the first two scenarios.

     

    Here are three samples of Notes documents from our standard demo database that I converted to Word:

    No Template: [White Oak.docx]

    On Quest Letterhead: [Eastern White Pine.docx]

    On Data Sheet Template: [Swamp White Oak.docx]

     

    Using This Feature

    When designing Target Data Definitions in NMSP, you can now add target columns of type WordDocument. You must also specify a column Name (which is used in the field mapping process).

    image

    Specifying a Template is optional. If you do not specify one, you will always get plain word documents containing whatever rich text contents and standard properties you choose to map from Notes. If you want to specify a template, simply click on the Template property and press the details button to launch the Word Template Options dialog.

    image

    In the Word Template Options dialog you can optionally import a Microsoft Word 2007 / 2010 template (.DOTX file) using the Import button. Note that a complete copy of the imported template will be saved as part of your Target Data Definition and will be used as the basis for any documents that you generate with it.

    Word Template Options dialog also includes a complete list of the available Mappable Fields, which are the parts of the generated Word documents that you might want to map Notes data to. This list includes the main rich text Body of the document, all the Standard Properties available in every Word document (Author, Created date, Subject, Title, Keywords, Category, Status and Revision) and well as any Custom Properties that may have been defined by the Word template you loaded.

    If desired, you can customize how these Mappable Fields appear on the Mapping tab. (Recall that Notes Migrator for Sharepoint maintains the distinction between the reusable Target Data Definitions that describe the schema of your SharePoint targets and the mapping of source columns to target fields in a specific migration job.) You can uncheck certain Mappable Fields that you do not want to show on the Mapping tab. You can also override some of the Mappable Field properties such as the MappableName that is visible on the Mapping tab, the AutomapNames property that provides a hint as to which Notes source columns should automatically map to the target field, and the AllowMultiple property which controls when mapping of two or more source fields to one target field should be allowed.

    Note that your Target Data Definition may well contain additional target fields that are not part of the generated Word Document. In particular, you can add target columns for any additional metadata properties that should be written to the SharePoint document library, rather than inside the generated Word documents (as described in scenario #3 above).

    You may also want to specify Folder names as well as the alternate locations for embedded attachments and OLE objects that should be migrated separately to SharePoint. These are existing features that are described elsewhere, but they apply equally well to Word documents.

    image

    One final thing you may wish to enable in your migrated Word documents is the new “Migrate Attachment Icons” feature. While this is not always desirable when migrating to List Items, it looks pretty nice in Word Documents.

    image

    Once you save your Target Data Definition, the various parts of the WordDocument field you defined will be available on the Mapping tab. Note that a special “Field.Part” notation is used here. In the example above we called the WordDocument field “Doc” so the mappable parts are called “Doc.Body”, “Doc.Title”, “Doc.Status” and so on. There is also a special field called “Doc.FileName” that allows you to set the names of the generated Word files from dynamic Notes data.

    image

    In most cases, you will (at a minimum) want to map the Html version of your Notes documents to the Doc.Body field and map the Subject (or a similarly descriptive Notes item) to Doc.FileName. You can also add additional mappings for standard properties and custom properties as needed.

    When you run the job, Notes Migrator for SharePoint will generate one Word document for each Notes document. You can inspect the migration log for any issues.

     

    Render with Form

    With Notes Migrator for SharePoint 5.3.3, it is now possible to “Render” a Notes document with its original Notes form. Rendering basically means taking a snapshot of the way a document is supposed to appear in the Notes client, and putting the entire thing into a single rich text field. This includes all the visual elements on the form, including form layout, field labels, graphics, and even “computed for display” fields. This can be a nice alternative to migrating all the individual data elements of a complex Notes application, especially in cases where you just want to archive the content from old applications and do not want to invest effort into migrating the application’s functionality.

    Here is an example that will hopefully make things clear. Consider a standard document library in Notes. As shown below, a document in this library consists of many data fields and the form used to display such a document shows it all in a nice layout.

    image

    The most obvious choice would be to pull out fields you wanted and map them to similar fields in a custom list, a wiki page, a Word document or some other type of SharePoint document. The example below shows mapping just the Subject, Category and rich text Body field to SharePoint (along with the created/modified metadata).

    image

    If I also wanted to grab the Originator and Reviewers data I could have done a little more work to get these as well. If my Notes database had many different forms for displaying different types of documents, I would have to start thinking about how to handle them. Perhaps I would start using Content Types or split the data into multiple lists. If I wanted to keep the “look and feel” of the original form layout that we had in Notes, I could start customizing my form in SharePoint Designer or InfoPath. All of this is certainly possible, and Notes Migrator for SharePoint contains many advanced features for making all this as easy as possible, but sometimes it is hard to justify the effort. We especially see this with old, out-of-date applications. Sometimes our customers don’t care about rebuilding a working application in SharePoint, they just want to archive the data in a user-friendly format.

    Our “Render” feature is intended to solve this problem. Here is an example of the above document rendered with its original Notes form and saved as a single rich text field. We did not have to figure out what the various fields in the application did; we simply captured the way it used to look in Notes and put it in the Body field.

    image

    Notes Migrator for SharePoint 5.3.3 contains several new features for rendering Notes documents.  If you want to skip the boring details of defining a source data definition for rendering the documents in your application, you can start by loading in the prebuilt “Render.pmsrc” data definition included with the product.  Or you can customize your own data definitions as described below.

    First of all, there is a Render target column type that you can use in your Source Data Definitions.  In most cases you will want to set the Option to HTML for mapping to SharePoint. Specifying a Form to render with is optional.  If you don’t specify one, each document will be rendered with its default form (using the “FORM” item in each individual Notes document). 

    image

    Images, objects and attachments from the form may be included in your migration job by using the “Scope” properties in those source data definition columns.  Again you can specify an optional FormName property. This should match the FormName you specified with the Render columns (above).

    image

    On the mapping tab, you can map the Render column and attachment/image/object columns described above to any in a SharePoint list or page in SharePoint, just like you would have done with a rich text Body field.

    image

    Note that there are limitations with form rendering.  As a general rule subforms, computed subforms, hide-when formulas, and computed text all work fairly well. However, some complex structures (for example hide-when formulas inside the subforms) will not always resolve correctly.  In some cases it may be necessary to develop a simplified form to use while rendering documents. You can override the default form in the source data definition columns.