Lotus Notes to SharePoint Blog

Blog about Dell's Notes Migrator to SharePoint tool and other things related to Lotus Notes migration projects

SharePoint for All has moved

Dell Software is back at the SharePoint Conference in a big way!

Dell Software is back at the SharePoint Conference in a big way! As you know,
we’re always a premier sponsor of this event because its the #1 SharePoint
event worldwide, helping us deliver our SharePoint solution message directly to
our customers. This conference takes place March 2-6, 2014 at the Venetian
hotel and casino.

Image

LOCATION: Booth #1624 at the front of the expo hall, next to the social media lounge

SPONSORSHIP: We’re once again a premier sponsor with an extra big booth and signage in every other elevator in the Venetian and Palazzo – you’ll know we’re here!

ZANY GAME/CASH PRIZES: Expect another wacky game that lets you win cash and learn about our offerings

SURPRISE GIVEAWAY: Walk the talk with a really fun give away (stay tuned for forthcoming details)

OFFERINGS/EXPERTS FOR Q&A: We’ve got migration, customization, management, infrastructure and services experts on hand to answer your questions

SPEAKING SESSIONS:

  • Sunday, March 2
    • 8:15 p.m.-8:30 p.m. – The Dell SharePoint story @ Partner Theatre in expo hall
  • Monday, March 3
  • Wed, March 5
    • 12:30 p.m. – 1:00 p.m. – Best Practices for Connecting Business-critical Data into       SharePoint @ SharePoint Pavilion in expo hall

Final post from Steve Walch

Hi everyone.  As many of you know, the reason for my long silence on this blog is that I resigned my position as product manager and left Dell on June 30.  So I am no longer Notes-to-SharePoint guy.

Many people have expressed how much they valued this blog as a source of straightforward product information and as a library of practical tips, tricks and product walkthroughs.  Therefore, in the hopes that all this can continue, I will be handing over control of this blog to the new product manager Randy Rempel.  This will be my last post here.

Before joining Dell, Randy was a partner and Notes Migrator for SharePoint power user, so I am confident that the products are in good hands going forward.  I am told that work on the next release is underway as well as some exciting new developments for Social Migrator for SharePoint.  As before, the Dell community site (http://communities.quest.com/community/sharepointforall/notes) is a great place to get additional release information, post questions, etc.

When I think back on my time at Quest/Dell, the part I miss the most was helping all my wonderful partners and customers.  Whether in person, by phone, by web meeting, or by email, contact with the people actually using the tools to solve real migration problems was always a highlight.  I am glad for this last chance to say goodbye.  Happy trails!

 

Q&A from Tuesday’s webcast

Sorry we could not get to all your questions live.  Here is the entire list of questions:

Q:  Is this event being recorded?

    A:  Yes, you can view the recording now at https://www.quest.com/common/registration.aspx?requestdefid=51835.  Sorry about the recording level – be prepared to crank the volume up.

Q: If a Notes application exists for archive purposes only – what type of data migration should occur to get off of Notes but keep the data for a period of time. I’m not sure if SharePoint is the right platform in some cases but just want to get your thoughts.

    A: Render with Form is the most common approach. See this post: http://notes2sharepoint.org/2010/11/12/render-with-form/

Q: What QuickR versions do you support?  R8, or also back to QuickPlace?

    A: All versions of QuickPlace and QuickR

Q: Can you speak to how often you have companies using straight out of the box (OOtB) templates from Lotus?  Rather, out of x number of apps, how many of those apps are OOtB? 

    A: Varies widely but maybe 30% apps are based on standard templates – doc libraries, discussions, team sites

Q: If you “skip all the custom code that those crazy notes developers put into place”, do you also have a service offering to help write re-training documentation/collateral?

    A: Dell and Quest PSO teams offer these services, as do our awesome partners.

Q: I’d love to hear more about “Find unused databases”.

    A: Our tool helps you find the last usage dates across all replicas of each database. Simply sort by date and pick whatever cutoff you want to use.

Q: what are the benefits of migrating to SharePoint vs. say Adobe Lifecycle?

    A: That’s a question for Microsoft. I’m just the humble migration guy.

Q: How do you do the design snapshots? 

    A: We basically build a mini-template – copying all the design elements to a local database in a special location.

Q: What’s the minimum access level on Domino end and on SharePoint 2013 (online) end?

    A: On Domino, you just have to be a reader. On SharePoint, it depends on what you want to do. To write docs, you need write access. To provision lists, sub-sites, groups, user access you need to be a site administrator.

Q: Does the free analysis tool support Office 365?

    A: Yes. Or maybe “Not applicable” is a better answer. Analysis just accesses the Notes stuff.

Q: How to map Domino user id to Active Directory accounts for permission migration?

    A: We give you a variety of options, ranging from a directory lookup (if there is some property in the Domino Person document that SharePoint will accept as a unique identifier) to a brute force XML file.

Q: Can your migrations also retain functionality like Domino Attachment and Object Store (DAOS)?  I know this is not a db design feature, but a Domino Server level feature.

    A: No, but I believe that this is all transparent to the Notes APIs we call. So it should all “just work”.

Q: Are the OOtB templates, you have considered in your design of this too, only from Lotus, or also from Vendors? 

   A: Only templates from Lotus. Really, however, designing jobs for new custom Notes templates is not that hard once you get the hang of it. The hard part is when you need to develop a new SharePoint template with complex functionality (i.e. something to migrate the content to).

Q: Do your tools have the capability to “decommission” a Notes app once migrated?  For example, either delete it wholesale, or alter the ACL, or add page that redirects the user to SharePoint?

    A: No, we decided never to modify the Notes environment. We get more trust from our customers that way, but it means an extra step from you.

Q: Do your tools also take Roles into account for the ACL, form/view access, document access, etc.?

    A: Yes, we have the option of generating SharePoint Groups from Notes Roles. We use these groups in document-level security. We do not attempt to migrate view/form access.

Q: Can lotusscript or fomulas be migrated?

    A: No.  That is usually a manual step, but it is usually best to switch over to OOtB features and declarative workflow anyway.

Q:  Does the “add to lookup list” generate a warning?

    A: Not a warning, but I think we would log adding new items if you have verbose logging enabled.

Q:  What is the specific improvement for doc with one attachment?

    A:  See this blog post:  http://notes2sharepoint.org/2013/04/01/filter-by-rich-text-content/

Q:  I would appreciate a follow up call to discuss the questions I posted to the panelists.

    A:  If you would like to talk about our products in greater detail, your Quest/Dell sales rep will be happy to arrange that.

PS:  A good place to post additional questions is in our SharePoint Notes Migration community site:  http://communities.quest.com/community/sharepointforall/notes

Public webcast later today

Important notice regarding SharePoint 2013 support in Notes Migrator for SharePoint

When we shipped Notes Migrator for SharePoint 6.2 last month, this included support for SharePoint 2013 but we realized that had one very unfortunate limitation: 

The Import Service did not support SharePoint sites that used Claims Based Authentication.  This was actually not a new limitation, and it impacted a handful of SharePoint 2010 customers, but it suddenly became very important in SharePoint 2013.  The reason is that all SharePoint 2013 sites use claims by default.  So this effectively meant that most SharePoint 2013 users would be forced to migrate via the native SharePoint web services.  Not a terrible option, but definitely not as fast.  (Migrating via the Import Service can be 3 to 5 times as fast, especially if you have large attachments and use the Share Files Folder feature.) 

Happily, we have now resolved the limitation.  Starting with hotfix build 6.2.0.1060, The Quest Import Service will work as expected even with the new SharePoint 2013 defaults.  This fix will appear on our web site in our next point release (6.2.1), but many customers will not want to wait for that.  Therefore I would encourage anyone who wants to use the Import Service on SharePoint 2013 to proactively contact Quest support and request build 6.2.0.1060 (or any hotfix build after that).   

Filter By Rich Text Content

In SharePoint, document libraries contain files. Every “document” is a binary file that you can download to your hard disk, etc. Notes document libraries are more flexible however. Some documents contain just one file attachment, while others may contain lots of rich text (and possibly multiple file attachments) in a rich text “Body” field. In fact, it is common to see mixed usage patterns in one document library, making it difficult to figure out the best migration target.

In the past, the best approach has been to migrate document libraries to a SharePoint list instead of a document library. SharePoint Lists mirror the flexibility of Notes documents well and allow you to capture your rich text (if any) in a “Body” field and migrate zero, one, or many attachments to the list’s attachment area. Unfortunately, you lose all the advantages of SharePoint document libraries with this approach. In the cases where you know that most documents are really just single file attachments, you would probably prefer to migrate those directly to a document library.

What many migration teams want to do is apply the following policy for document libraries:

  • For documents that contain just one attachment (and no other rich text), migrate the attachment directly to the SharePoint document library with all the appropriate security and metadata.
  • For documents that contain Notes rich text, generate a Word or PDF document and place it in the same SharePoint document library with all the appropriate security and metadata.
  • For documents containing neither attachments nor rich text, either skip the document or create a stub entry in the target library.

In order to implement this policy, Notes Migrator for SharePoint 6.2 now includes a new record filtering option for Notes and Domino.Doc data source definitions.

image

On the record selection tab, check the “Select documents based on Rich Text Content” checkbox. This will enable a Details button where you can specify further details. First specify one or more rich text items you would like to inspect. Second, specify the criteria you would like to use for filtering documents:

  • Whitespace only
  • One attachment only
  • Multiple attachments or other rich text

This new record selection option allow you to create multiple migration jobs for each document library, each one implementing one of the rules in the above policy. Remember that the Notes Migrator for SharePoint migration console makes it easy to sequence multiple migration jobs for one database, and to automate these jobs for many databases of the same type.

Also note that we are planning a similar feature for extracting QuickPlace and QuickR folders, but this is not available in the current build.

Enhanced support for Lookup Fields in Notes Migrator for SharePoint 6.2

In version 6.0 we added support for migrating content to SharePoint lookup columns. In version 6.1 we added provisioning of new lookup columns. Now here are two new advanced choices for using Lookup columns to help you design more powerful migration jobs.

Add Missing Choices

Normally when you migrate content to a SharePoint List that contains a lookup column, the Notes Migrator for SharePoint code tries to find the data value and correlate it a record in the lookup list. For example, you may be mapping a VendorCode item from your Notes data source to the “Vendor Code” column in your SharePoint target list, which is lookup column that connects to an indexed Vendor Code column in a list containing Vendor records. If you set the new “Add Missing Choices” property in your column definition to true, we will add a new record to the Vendors lookup list if one does not already exist. (This is now very similar to how “Add Missing Choices” works for Choice columns in our tool.)

image

Note that in order for this to work, the Lookup list should have reasonable default values for all the other columns (especially the ones that are required columns).

Override Lookup Column

A mapping such as the one described above (mapping a VendorCode item from your Notes data source to a Vendor Code column in your lookup list) is fine for simple cases. But sometimes the situation is more complicated than that, and you need to use some other value from the Notes data source to correlate the data.

Suppose for example that the Notes application used VendorTaxID to link things together but the new SharePoint template is designed to use Vendor Codes. With version 6.2 you can now specify an “Override Lookup Column” property to control how Notes Migrator for SharePoint locates the record in the target list to link to. In the following screen shot, note how the Tax ID column in the lookup list overrides the Vendor Code column that is configured in SharePoint. Now you can map VendorTaxId values from Notes and get Vendor Codes in your SharePoint lookup column in the end.

image

Another great use for this new feature is the case where the Notes application used response documents to link two types of records. For example, it is common to use response documents to link user Comments to a patent document, such as a team’s Action Items (shown below).

image

In order to correlate the Comment documents to the parent documents, we need to use the internal Notes universal IDs (a.k.a. UNIDs). As luck would have it, Notes Migrator for SharePoint always stores the UNID of each migrated Notes document in a hidden “NotesUNID” column in SharePoint. So you can use that column as your Override Lookup Column and simply map the parent IDs to the lookup column:

image

Note: this technique works perfectly with the SharePoint Blog site template. The Comments list has a lookup column that references the Posts list. Now you can correlate Comments with Posts using internal Notes UNIDs, which is a lot more reliable than using Titles (which may contain duplicates).

Direct Folder / Document Set Migration

In previous versions of Notes Migrator for SharePoint, users could map document metadata (for example the Category property of a Notes application or the {BinderName} property for Domino.Doc documents) that would cause folders to be created in SharePoint. Folders would be created as needed as documents were being migrated. This worked for most cases, but there were limitations that customers would occasionally ask about:

  • Since we only migrate folders as a “side effect” of migrating documents, there was no way to migrate empty folders.
  • Similarly, there was no way to create the folders ahead of time (before migrating the documents)
  • There was no way to set permissions, created/modified metadata, or additional data columns on the newly created folders

Now in Notes Migrator for SharePoint 6.2 we offer a way to do direct folder migration. This is really two separate features that work together…

Migrating records to folders

image  image

On the Advanced tab of your Target Data Definition, you can now indicate that you want to migrate to a folder in your target list or library.  In this mode of operation, every record you extract from the data source will result in a folder being created, instead of a document!  The only additional requirement is that you map at least one item to a target column of type Folder (which controls the new folder names). Many of the usual document migration features will now apply to folders including:

  • Mapping of permissions (using the “Map Reader/Author fields” checkbox on your Advanced tab)
  • Mapping created/modified metadata to folders (using the “Preserve Created/Modified” checkboxes on the Map Data tab)
  • Mapping additional data items to folders (requires creating a new Folder content type). 

Note that many job features that would apply to document migration will not apply to folder migrations. For example, document generation and duplicate document handling options would be disallowed in in this context.

Extracting information from Domino.Doc Binders

image image

One of the things that customers clearly want to do with this new feature is to migrate all the information from their Domino.doc Binders to SharePoint folders. To support this, we have added a new option to do exactly that in Domino.Doc Source Data Definitions. Simply check the Binders radio button on the Document Selection job, and now you are extracting Binders instead of Documents. Each row in the Preview represents a Binder in the current file cabinet and we have included additional columns for all of the standard Binder metadata available in Domino.Doc. Of course you can add additional columns to this query as well.

So putting these features together, you would typically map the {Title} property of your data source to a Folder column in your target. Simply checking “Map Reader/Author fields”, “Preserve Created/Modified identities”, and “Preserve Created/Modified dates” should bring over most of the other metadata but you can certainly add additional mappings if desired.

Note that this feature will only write new SharePoint folders; it will not update existing ones with the same name. So a best practice is to run the Binder migration job first (to create the folders with all the properties intact) and then run you normal document migration job.

Also note that we are planning a similar feature for extracting QuickPlace and QuickR folders, but this is not available in the current release.

Migrating to Document Sets

Similar to migrating to folders, Notes Migrator for SharePoint 6.2 also give you the ability to migrate directly to document sets. The situation here is very similar to what was described above. The tool already allowed creation of document sets as files within those document sets were being migrated. This is a powerful and popular feature, but suffered some of the same limitations.

  • Since we only create document sets as a “side effect” of migrating documents, there was no way to create empty document sets.
  • Similarly, there was no way to create the document sets ahead of time (before migrating the documents)
  • There was no way to set permissions or created/modified metadata on the newly created document sets separately from the documents.

The solution is similar to the folder solution described above. On the Advanced tab of your Target Data Definition, you can now indicate that you want to migrate to a document set in your target list or library.  In this mode of operation, every record you extract from the data source will result in a new empty document set being created, instead of a document!  The only additional requirement is that you add a target column of type DocumentSet and map a value to the DocumentSet.Name property. All of the other features of document set migration (described here) still apply. The difference is that every record you select gets mapped to a document set instead of a file within a document set.

Automatically ZIP file attachments

Many customers have requested the ability to compress Notes file attachments while migrating them to SharePoint. There are a number of good reasons for wanting to do this:

  • Save disk space on SharePoint server
  • Get around SharePoint file restrictions (i.e., blocked file extensions and/or size limits)
  • Improve the bandwidth sending data to remote SharePoint servers
  • Eliminate problems (hangs and memory leaks) when embedding certain types of file attachments inside Word documents

Notes Migrator for SharePoint 6.2 now allows you to achieve this in Notes and Domino.Doc migration jobs via a new property on Attachment columns in your Source Data Definition. The “Compress” property may be set to “None” (the default) or to “Zip”.

clip_image002

You can also configure a set of global exceptions to this rule on the Notes tab of the tool’s Options dialog. The “Compression Exclusions” option allows you to specify any file extensions that should never be zipped. This would typically include media files that are already well-compressed and would not benefit from zipping.

clip_image004

When the Compression property is set to “Zip”, all extracted attachments will be compressed and placed inside a ZIP file when migrated. As shown below, any icons or text links to the attachments will appear the same as before, but when the user clicks on them they will open up a ZIP file instead of the “raw” file.

clip_image006

Additional Notes

Zip files will always be excluded from further zipping, even if they are not specified in the “Compression Exclusions” list.

If you wish to use this feature to migrate files that were previously blocked by SharePoint (for example .EXE files), be sure to remove these extensions from your Blocked Files list on the SharePoint tab of the tool’s Options dialog.

When using this feature with Domino.Doc sources, add a custom Attachment column as described above. Map this custom column (instead of the predefined {Attachments} column) when mapping attachments to a target column.

clip_image008

This feature is not currently available to QuickR or QuickPlace.

Attachments in generated Word documents… use caution!

We have had a number of support cases on this over the last year, and we have made an effort to clarify our position on this in our latest release notes:

Migrating file attachments inside of MS Word documents is not recommended for large migration jobs. Since Word attachments are implemented as “Packager” OLE objects, the migration process is forced to invoke Packager code as well as the OLE handlers for the file type in question (often in separate processes). The problem is that each type of attachment is handled differently depending on which type of application created the attachment. So (for example) the first 1000 attachments may work fine and then document 1001 has a different type of attachment that causes a memory leak when Microsoft converts it into a Packager object. When migrating attachments and OLE Objects to embedded objects in MS Word, it is highly recommended that the workstation performing the migration has native applications installed that can open and edit every type of attachment that the migration jobs will be encountering during the migration.

The two recommended workarounds are:

  • Migrate attachments separately to the SharePoint document library. (The links from the Word documents to the attachments will be preserved.)

  • Place all attachments inside ZIP files inside the Word documents. (This is a new feature for version 6.2.)

If you still insist on embedding attachments into Word documents, you should understand that you are doing so at your own risk and that Quest support will probably not be able to help you if problems occur. Here are some tips that have helped other customers in the past:

  • There are known issues with packaging Adobe Acrobat 10 objects. If you have PDF attachments, be sure to install Adobe Acrobat Reader version 9 on the workstation.

  • Packager leaks usually have to do with cases where packager cannot use type-specific COM objects (including all .MSG files, and including .PDF files when the PDF reader is not installed)

  • Leaked resource handles accumulate per Windows process. This means that you should restart NMSP Designer between large jobs, even if they are successful. It also means that batches of jobs in the NMSP Migration Console would still not be a good idea.

  • There may other issues with packaging certain types of objects. Since we cannot product how third-party OLE handler will operate, this is beyond our control.

Introducing Notes Migrator for SharePoint 6.2

This release will be downloadable from the web site in a couple days. 

The biggest feature by far is full SharePoint 2013 support.  Everything you could do in SharePoint 2010 you can now do in SharePoint 2013 and the new version of Office 365.  Its that simple!

The other features from our release notes are:

  • Zip attachments while migrating
  • Support for Pass-through HTML
  • Special handling of documents containing just one attachment
  • Full folder migration (example: Domino.Doc binders)
  • Support for multiple Notes passwords
  • Improved lookup field support (add missing choices)
  • Improved support for migrating to Claims-based environments
  • Render with form improvements (computed subforms, etc.)
  • Discovery process is more robust

Some of these features will be more self-explanatory than others.  I will be posting detailed walkthroughs of a lot of these features here in coming days.  You can also check out the “what’s new” section of the user guide or watch my recorded “update” training here: http://communities.quest.com/docs/DOC-14812

Checklist for planning the migration of a Notes Database

Before migrating a Notes database to SharePoint, there are a variety of details to gather and choices to make. Thinking through the items on this checklist will help ensure a smooth migration process with minimal surprises.

Source database details

  • Source type (Notes, QuickPlace, QuickR, Domino.Doc)
  • Source name
  • Notes template used (if any)
    • Level of customization from template
  • Domino server & path (closest replica to migration team)
  • Special Notes identity required to access database (if any)
  • Public keys needed to decrypt documents (if any)
  • Approximate number of Notes documents to be migrated
  • Approximate size of Notes content to be migrated
  • Desired priority of migration
  • Desired phase/group of migration (intended to capture clusters of databases that should logically be migrated together)
  • Documents that should / should not be migrated
  • Unique “document types” to be migrated (this usually correlates to distinct Notes forms that back existing data documents)
  • Design Details that will need to be accounted for
    • Forms
    • Workflow
    • External Connectivity
  • Third Party validation/security auditing
    • PCI Compliance
    • FDA Compliance
    • HIPA
  • Security and permissions that will need to be maintained and updated
  • Archiving of data
    • Date Range
    • Data Type

Target Site Details

  • SharePoint site URL
  • Account to be used or provisioning and content migration
  • New or existing site?
    • Template to use for provisioning site (may be custom developed)
    • Site provisioning details – parent site, inherit permissions
    • Provision site security (user permissions, group permissions, roles)?
    • Are specific content databases required?
  • Source of identity mapping information
    • Users
    • Groups
  • Out-of-box SharePoint features to enable/leverage
  • Do file size limits and blocked file types need to be changed from the defaults
  • Site Columns in use
  • Content Types created
  • Workflow type and creation
  • Search and Indexing

Target List/Library (specify for each document type if appropriate)

  • Target list or library name (Naming schemes in use)
  • New or existing list?
    • Template to use for provisioning list (may be custom developed)
    • Add to Quick Launch
  • Provision list/library security (user permissions, group permissions, roles)?
  • Allow tool to upgrade existing schemas if needed?
  • Desired folder structure within list or library
  • Map created/modified identities/dates
    • Behavior if user mapping fails (fail or substitute default user)
  • Out-of-box SharePoint features to enable/leverage
  • Third Party solutions (templates, web parts, etc.) in use or available

Mapping details (specify for each document type if appropriate)

  • Content Type for migrated documents (if any)
  • Target document format
    • List Items
    • Extracted binary file attachments only
    • Generate InfoPath docs (which template; target Forms Server)
    • Generate Word 2007 docs (which template)
    • Generate Adobe Acrobat (PDF) documents
    • Generate Web Part pages (which template)
    • Generate Content pages – Publishing pages, Basic pages, Wiki pages
    • Generate HTML files
    • Generate MIME files
  • Use separate library/folder for attachments, images, embedded objects?
  • Field mappings
    • Source data item/column names and types
    • Data transform required (example: translation of keyword values)
    • Target data column names and types
    • Using Lookup columns?
    • Using Managed Metadata columns?
  • Migration options
    • Map document level security (readers / writers)?
    • Convert DocLinks to dynamic links (i.e. use the Link Tracking Service)?
    • Map discussion/response hierarchies
    • Map calendar logic (repeating meetings)
    • Map approval status / workflow state? (How expressed in Notes app?)
    • Map version histories (How expressed in Notes app?)
    • Map data in date ranges
    • Render with form (archive only)

 

NOTE: Two important items were not covered in the above checklist:

  • Development work that may be needed in SharePoint Designer, InfoPath, Visual Studio to reproduce the custom logic you had in Notes.
  • The opportunity to reduce your migration effort by consolidating similar application designs, reusing custom development work, and automating large scale migrations.

Both of these topics are discussed in detail in the Quest white paper “Migrating Lotus Notes Applications to Microsoft SharePoint: Understanding Application Complexity and the Value of Consolidation and Automation” which may be download from http://www.quest.com/documents/landing.aspx?id=9746.

Still some confusion about Dell Software

In case you never got the memo, Dell acquired Quest Software back in September (link).  Since then the organization formerly known as Quest has been on a rapid growth path to becoming a major pillar of Dell’s new software division.  Yes, Dell does software now, but it is probably going to take a little while for everyone to get the idea.

At the November SharePoint Conference (SPC) the usual Quest sponsor booth was now a fully branded Dell booth.  I was working that booth and saw a lot of confused expressions as people tried to grok our signage.  Some were ignoring the booth because they did not need any laptops.  Others assumed that if Dell had SharePoint tools, they must be new 1.0 products.  Once we explained things, people were generally reassured that the Quest tools they knew and loved were alive, well, and likely to grow faster than ever.

So please bear with us as we proceed slowly and thoughtfully with rebranding of existing tools, etc.  Notes Migrator for SharePoint 6.2 will go out with Quest branding next month, but my new product Social Migrator for SharePoint (link) will go out with Dell branding.  Exciting times!

Online Documentation is live!

The full Notes Migrator for SharePoint documentation set installed with the product (User Guide, Installation Guide, Release Notes) is now also available online at  http://documents.quest.com/Product.aspx?id=20.  This will become a standard for many Quest products, but I am happy to say that we were one of the first!

One thing I like about this new facility is how the search works.  Depending on where you start from, you can search one particular document, all Notes Migrator for SharePoint documents, or all Quest documents.

Allowing PDF files to open in the browser

Does your SharePoint server force you to save PDF files to disk first, instead of opening them directly from your list or library?  If you are lucky enough to have access to SharePoint Central Administration, there is a simple way to fix this in your web application’s General Settings.

 

The details are discussed in this blog post by Bram Nuyts: http://bramnuyts.be/2011/11/03/allowing-pdf-files-to-open-in-the-browser/

Now if only Microsoft would change this setting on Office 365!  Without it, archiving your old Notes applications as PDF documents is a lot less satisfying.

Exporting attachments and generated files to the file system

In case you missed it:  in Session 6 of my Partner Training webcast series, I introduced a free utility for reorganizing the intermediate files that Notes Migrator for SharePoint writes to disk.  Not everyone was clear on what such a utility was good for, so I thought I should elaborate:

Every once in a while, we hear one of theses requirements from customers:

  • Extract all the attachments from a Notes database and place them in the file system (instead of SharePoint)
  • Convert Notes documents to Word Documents and place them in the file system (instead of SharePoint)
  • Convert Notes documents to PDF Documents and place them in the file system (instead of SharePoint)
  • Convert Notes documents to InfoPath Documents and place them in the file system (instead of SharePoint)

The trick to doing any of these things is to use our intermediate data file format.  Just design a Notes Migrator for SharePoint migration job and when running the job, select Save to Intermediate File.  This will give you the option of saving any attachments or generated documents as disk file rather than encoding them in the main XML output file.

image

The problem with this approach is that the files that are saved to disk are not really that useful because they are arranged by Notes UNID, not any way that would be useful to humans.  That’s where the free utility come in.  This utility, which may be downloaded from our Notes migration community site, will take all these files and place them in the folder (or network share) of your choice and arrange them according to your requirements.  If you want to see an example of this tool in action, watch the video of Session 6.  Slides starting at about minute 77 and demo starting at minute 91.  Have fun!

Notes Migrator for SharePoint and Quest Web Parts collaborate to deliver support for migrating Notes Sections

The best kept secret in Notes Migrator for SharePoint 6.1 is that we added full support for migrating Notes sections.  That’s right, I’m talking about the expand/collapse sections with the little triangles in Notes rich text items.  When we migrate any rich text that contains such sections, we generate DIV tags for the section headers and section bodies in the resulting HTML. 

The problem, of course, is that these tags do not do you much good because, by default, SharePoint does not know how to use them.  That’s where Quest Web Parts for SharePoint version 5.7 comes in.  If you place our rich text web part on your page, all those migrated sections suddenly start working!  Even better, users can continue to create expand/collapse sections in new documents.  Won’t your Notes users love that?

image

See this post on the new version of Quest Web Parts for SharePoint.  If you scroll to the bottom, you will see a section “Improved Integration with Notes Migrator for SharePoint” and a really cool video describing what is going on.

Quest Web Parts for SharePoint has always been popular with Notes customers.  Features such as multi-level threaded discussions, dynamic navigation areas, and tabbed form layouts has been helping migration teams satisfy the requirements of their migrated Notes applications (at a drastically lower cost than hand-coding those features) for years now.  Of course it is not just for migrated Notes apps, but is really useful of many other SharePoint development scenarios.

Do you require more in-depth design analysis?

After you have completed your Notes application analysis with Notes Migrator for SharePoint, you will probably discover that some of your complex Notes applications will require custom development on SharePoint.  To help you clarify the level of development required and assist the developers performing the actual conversion, we recommend Teamstudio Analyzer for the in-depth design analysis.  Click here for more information on how Quest partners and customers can benefit from using Teamstudio Analyzer.

Notes Migrator for SharePoint 6.1 Reviewers Guide

Notes Migrator for SharePoint 6.1 is a significant product release that pushes the product further in three important areas:  Design Migration, Content Migration and Pre-Migration Analysis.  This release will be “Generally Available” on the Quest web site in a few days.  If you were in the beta program you already have the final build (6.1.0.626).  Below is a high level list of this release’s most important new features.

Design Migration

Migrate Notes Views to SharePoint Views – This feature will allow you to select a Notes view and generate a similar view in any SharePoint list or library.  Because the mapping of columns in your view is intimately tied to the mapping and provisioning of data columns that occurs in your migration job, this new capability is also tied to your migration job.  Specifically, view migration is found on the new Advanced tab of your Target Data Definition.  You can design new views there and the views are provisioned when the migration job is run.  A nice side effect of this model is that you can run the same migration job against many new or existing lists or libraries.

clip_image001

clip_image002

Press “Import from Notes” and then select the Notes view you want to migrate.  The view migration wizard will then attempt to design an equivalent SharePoint view, mapping as many Notes columns as possible to existing SharePoint columns OR to new columns defined in your job’s target data definition.  As you might expect, there are a number of things that can go wrong with such an automated mapping.  The two most likely issues are (1) the Notes data item shown in the view had not been migrated to the SharePoint list yet or (2) the Notes view column is computed and the formula is too complicated for a tool to convert.  For issues like this, manual intervention is required and the migration wizard guides you through that.  In the View Columns step, it gives you a side by side view of the Notes and SharePoint columns and highlights the parts that it needs help with in red.  You can decide to manually map a Notes existing columns, define a new data column (which will be added to your migration job), or even specify a formula for a new computed column in SharePoint.  Similarly you can also specify the sort order, the grouping and the document selection rules, or you can just take the defaults. 

Content Type Generation – These feature takes the tool’s ability to migrate schema (select fields from a custom Notes application and provision a similar schema in a custom list) to a new level.  Now you can migrate your custom Notes application schema to SharePoint content types instead.  You can create new Content Type Definitions on the new Content Types tab of your Database or Technical Class records in the Migration Console.  You can design them from scratch or, more likely, you will generate the Content Type definitions from a Notes Form, from an existing migration job, or from an existing list that you have been working on.  (The last two options allow you to do your initial development and testing on a custom list and then transition to content types later.)

clip_image003

clip_image004

Once you have generated an initial Content Type Definition, you can further customize it, specifying the parent content type, the group, and the columns.  The columns editor looks like the Target Data Definition columns editor, but in this case you are defining Site Columns instead of List columns.  When you are ready to provision your new Content Type, the tool will try to find existing Site Columns that match your specification and will provision new ones if needed. 

clip_image005

clip_image006

New Column Provisioning Options – The ability to provision lists and libraries while migrating content has long been a popular feature.  This capability has been greatly expanded to allow provisioning a nearly complete list of SharePoint column options, such as default values and data validation.  You can even set columns to be read-only (something you can’t normally do in the SharePoint user interface).  Note that as before, the provisioning options only apply when the tool is first provisioning the columns; it does not upgrade existing columns.

clip_image007

Provision Calculated columns You can now define Calculated columns in your target data definition.  These will be provisioned in SharePoint when your migration jobs are run. This Microsoft documentation page explains the legal syntax for SharePoint formulas:  http://msdn.microsoft.com/en-us/library/bb862071.aspx.  You can also use our new built-in formula editor, which appears at several points in the product.

clip_image008

clip_image010

Provision Lookup Fields – Lookup fields can now be provisioned in SharePoint lists.  Previous versions of the tool allowed you to migrate to Lookup fields, but you first had to manually configure them in SharePoint. You can even configure lookups from lists in other sites in the same site collection (something you can’t normally do in the SharePoint user interface).

clip_image011


Content Migration

New Run Job options – During normal migration jobs, Notes Migrator for SharePoint first takes care of the provisioning steps (creating the list/library if needed, adding the appropriate content types to the list, adding any missing columns if needed, and creating views, setting the list/library permissions) and then migrates the content.  Now you can now choose to run just the provisioning bits without migrating the content.  This will be useful in cases where you want to review the resulting list schema and perhaps make changes to the list settings before migrating content.

clip_image013

clip_image015

Also note that the above screen shots show off the tool’s new Run Job button with the drop down menu of various modes of running the tool, such as importing data to and importing data from intermediate XML files.  Since this button is also available in the Migration Console, users can now leverage the intermediate file options from there (it was formerly available in the Designer Client only).  Similar functionality is also available in the tool menus and the bulk Migrate To SharePoint task in the Migration Console.

Migrate by Form Used – When selecting records from your Notes database, you can now elect to query records by form.  You can pick one or more Forms from the Notes database design or type them manually.  This will limit the records selected to include only those data records that were flagged with the forms you selected (i.e., where the “Form” item was set to one of those form names).

clip_image017

clip_image019

If you specify forms in this way, the Select Items and Columns dialog will also display the data items defined on your selected forms.  (Remember, however, that background agents, etc., may have also set data items on the Notes documents, so you may still need to look at the Sampled Items node to discover those.)

Improved Content Type Support in Migration Jobs – Notes Migrator for SharePoint already had pretty good support for dealing with content types while migrating content, but this release extends those capabilities and also makes it much easier to use.  The new Manage Content Types section on the first tab of the Target Data Definition allows you to push site content types into a new or existing SharePoint list.  Adding the content type to the list is a prerequisite to writing documents with those content types, so this new feature eliminates that manual step and make more automation possible. 

clip_image021

In addition, you can now associate each content type with one or more Notes forms.  If you do this, then the tool will automatically assign the content type for each migrated document without the need to do any explicit mappings.  (For more complex scenarios you can still explicitly map any piece Notes data, including the result of a formula evaluation, to a ContentType field.)

Note that this feature is completely independent of the ability to generate new Site Content Types, as described above under Design Migration, but you may often find yourself using the two in combination.

Normalize multi-valued items to multiple documents – It is common for Notes forms to implement detail records (the things that relational developers would have set up a one-many-relationship for) using multi-valued data fields arranged to look like a table, as shown below. Users would then fill in as many entries as the needed, making sure that the values from each multi-valued item lined up properly.

clip_image023

clip_image025

Now you can use Notes Migrator for SharePoint to extract the multi-valued items as separate records. To do this set the Multi-Value disposition option for the columns that you expect to contain arrays (ContactName, ContactTitle, and ContactPhone in the above example) to “Normalize”. This will cause NMSP to generate multiple SharePoint items (three in the above example) for the values in the columns designated for Normalization.

clip_image027

clip_image029

Note that if you also wanted “header” records you could first migrate those using a separate migration job. You could use lookup fields or some other mechanism to relate the “header” and “detail” columns. In the above example, Customer ID in the Contacts list is set up as a lookup field that references the Customers list (where Customer Name and Category are also stored).

Finally, you may encounter Notes forms that list out each item in separate rows, instead of using multi-valued items. NMSP can handle that case as well, but the migration team will need to do a little work with formulas to generate multi-valued columns to Normalize on.  For example, you could define a source data definition column with the formula “Product_1: Product_2: Product_3: Product_4” to generate a “Products” array.

clip_image031

Dynamic link tracking for URL columns – When migrating Notes data to SharePoint URL fields, you can now leverage the tool’s popular Link Tracking Service for those links (the same way you currently do for Doc Links in Rich Text Fields).  You do not need to do anything special to enable this.  Simply start migrating data items that store notes:// links (or formulas that generate notes:// links) and they will be converted to dynamic links that will ultimately point users to the migrated versions of those documents. 

As a reminder, the tool supports two input formats for Url fields.

  • <url> –  sets the url to be the display name in SharePoint
  • <url>, < name> –  sets the url and display name separately in SharePoint

Improvements in “Render With Form” function – This very popular feature, which allows you to extract the content of any custom Notes database (regardless of the complexity) so you can archive them as simple rich text documents in SharePoint, has been improved in a number of respects. Handling of dynamic subforms, computed fields, computed text, and keyword fields have been improved, resulting in even better looking rendered documents.

Provision Alternate Libraries – Users have often relied on the tool’s unique ability to migrate images and attachments to a separate location from the main documents being migrated.  For example, you might want to place all attachments in the Shared Documents or Site Assets document library.  Previously you would have had to provision these alternate libraries manually before running your migration job.  Now the tool provisions these automatically if needed.

clip_image032


Analysis / Migration Console

Capture Design Copies during Analysis – Now when you perform a Design Analysis on a set of databases you can elect to also create a local design copy.  For every database scanned, the tool will create a small copy of the database (design elements only, similar to a design template) on the local machine on which the design analysis is being performed.  The intent of this feature is to allow for consultants and other migration personnel to be able to view the full database designs even while they are disconnected from the production environment, which is particularly important on large analysis projects where in-depth manual design analysis is required.  The location of these databases is controlled by a setting on the Analysis tab of your Global Options dialog, and defaults to a folder in the tool’s <ProgramData> area.  Users can the easily open the Local Design Copy from any database view.

clip_image033 clip_image034

Classify by Last Used (All Replicas) – This enhancement expands the available options for automatically determining which Technical Class or Business class a databases belongs to.  Now you create a rule for recognizing class members based on when it was last used across all known replicas.  For example, you can create a class that groups all databases with a Last Used date greater than 356 days ago.

clip_image036

Import data into repository from CSV files – This enhancement allows users to read in records from a CSV file and update vales in certain Quest Repository database records.  The tool is available on the main Notes Migrator for SharePoint node.  First the user is prompted to select a CSV file and then a mapping dialog is displayed.  For each column in the CSV file, the user can choose to map it to one of the available database record properties.  Not all database properties can be imported from external data sources.  The intention here is that users who perform manual analysis/triage projects using spreadsheets or external tools can import that data back into the Migration Console.

This feature can also be used to add new database records into the repository (by Server and File Path, Replica ID, or Database Key).  The intention here is to allow users to import a list of databases to be analyzed or migrated.  Users would typically follow up such an import with a full analysis of those databases to populate the remaining database properties.

clip_image038


General

Performance and scalability for Migration Console – The database views have been re-architected to support 60,000+ databases at a time.  With a very large repository, users may experience a delay while starting the tool as all the records are loaded into memory.  After that, scrolling, sorting and filtering database views and opening additional views should be very fast.  Special protections have also been added to prevent you from running very large reports that are likely to crash the migration console.

New User/Group Mapping options – When performing user/group name mapping, the Output Translation option now allows you to further transform the name that results from your name lookup before submitting that name for resolution in SharePoint.  This may be useful when your environment requires specific name formats that are not immediately available in your mapping source.

Also, the Test User/Group Mapping tool now has a Validate in SharePoint button which will try to resolve the name that your configured mapping process produces so you can verify that it really works in SharePoint. This should make experimenting with and debugging various user mapping options a little easier for everyone.  Note that this capability is only available when using client-side user mapping (not when configuring server-side mapping in the Import Service).

clip_image039

Improved CBA/FBA support – Connections to SharePoint sites using Forms Based Authentication now automatically renew cookies as needed during long migration jobs and other operations.  The tool will routinely check the expiration time of the authentication tokens that it holds for the client and, if a token is due expire within a certain window, it will force a new authentication.  (The time limit will be 30 minutes, but this can be changed in your SharePoint Environment settings.)  Depending on your particular authentication system, this may appear to the user as a browser prompt forcing the user to re-authenticate.  In other cases, it may appear as a browser window that opens briefly and then closes again.

clip_image040

Windows Authentication using alternate account for Link Tracking Database One new connection option is available for connection to the Link Tracking Database.  You can now specify that you want to use Windows Authentication, but supply an account other than your own.

clip_image041

Bulk Editing of certain database / class / job properties – Expanding on the tool’s capability to select multiple databases and set properties in bulk, you can now set a number of additional properties in a large number of selected databases.  You can even set certain properties inside the migration jobs assigned to the selected databases.  Finally, you now have a similar set of options for Technical Classes and Business Classes as well.

Usability improvements A number of things have changed in the product to improve to make the tool easier to use, including rearrangement of dialogs and additions of menus.  More pop-up help icons and more context sensitive links to help topics have been added.  A completely rebuilt “Add/Remove Columns” dialog and the built-in documentation describing the 170 possible built in columns makes customizing views much easier.

clip_image043

clip_image045

Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: