Notes SharePoint Blog

Steve Walch's blog about his favorite migration tool and other things related to Lotus Notes migration projects

Monthly Archives: February 2011

All Database View/Report Columns

With each new release of Notes Migrator for SharePoint, we add new functionality and this usually means new database properties that you can view in the Migration Console.  For example, our new “Blocked / Oversized File Detection” feature gives you four new columns that you can add to any database view or report:

image

With version 6.0, we are now up to 172 columns that you can display for each Notes database in your environment.  These columns are set in a variety of ways:

  • Discovery
  • Data Analysis
  • Design Analysis
  • Usage Analysis
  • Automatic Classification
  • Automation via Class Rules
  • Manual Triage
  • Performing Migrations

While we document all the available database columns in the appendices of our User Guide, it still can be difficult to understand what columns are available to solve a particular problem or where a given piece of data comes from. 

I have, therefore, created a new spreadsheet that includes extended documentation and is sortable in a number of ways.  Download it here: [NMSP 6.0 Database View Columns.xlsx].

Extracting all the users from a set of databases

The Extract Database Users tool (new in Notes Migrator for SharePoint 5.3) allows you to select one or more databases in any database view in the Migration Console and then extract all the user names contained in those databases.  This tool is useful for simply gaining an understanding of the users involved in a group of Notes applications, but the primary purpose of the tool is ultimately to generate a user mapping file that NMSP can use at migration time.

clip_image001

Depending on how much analysis has been done for these selected databases, we may extract user names from the database ACLs, the Created By/Modified By metadata, the document level security, or the usage activity.  As explained below, users may also be added to the list by expanding Domino groups and by importing existing NMSP User Mapping XML files.  The sources of the user names are listed in the view columns shown below, and you can filter the sources shown to only list users that came from certain sources.

clip_image002

The type of user name (Person, Group, Unspecified) is also shown and you can filter based on them.  Note that Unspecified users may become specified as you perform certain operations such as group expansion or imports.  Finally, you can manually set the user type by using the combo boxes in the view. 

clip_image003

You can also filter by the Notes domain.  This is a simple text match against the last part of the abbreviated name, so either “Westford/IBM” or just “IBM” would select “John Smith/Westford/IBM”.  If is common to want to select all the users plus all the groups, which the “No domain” checkbox allows you to do.

clip_image004

If a group is listed in a database, it is sometimes useful to be able to find all the members of the group.  If you press the Expand Groups button, NMSP will contact the configured Group Resolution Server (from Advanced Configuration) and look up every Group and Unspecified entry (in case it really is a group).  Any new members will be added to the list and indicated as “ACL via Group”.

clip_image005

To remove users from the list, you can select one or more row using the selection column on the right (or press Control-A to select all of them) and press the Remove button.

clip_image006

The last column in the view is the SharePoint names column.  You can set these names automatically using the Import function, using the Set SharePoint Names function, or by typing them in manually. 

The Import process loads in users from existing XML User Mapping files and sets the Imported column.  Imported data is merged with existing data but if a SharePoint name is specified in the imported file, it will overwrite the existing name every time. 

The Set SharePoint Names function gives you several ways to automatically assign your SharePoint names. 

· Load users from Domino Directory – use any field in the user’s Person document on the Domino directory as the new SharePoint name

· Set Default using Format String – Generate a new SharePoint name by substituting the various parts of the Notes name.

· Set Default using the Notes common name  – Use the simple common name as the SharePoint name

Note that in all these cases, existing SharePoint names will be preserved unless the override SharePoint names flag is already checked.

clip_image007

Finally, you can press the Export button to generate a User Mapping file (either in an XML or comma delimited format).

clip_image008

MessageStats Report Pack for Lotus Notes

People love MessageStats.  In the three years since joining Quest, I have met many customers and partners who think of MessageStats as the Quest product, compared to which migration tools are a necessary evil to get a short-term project done.

What many people do not know is that MessageStats, which reports from a wide variety of email systems, includes a great new Lotus Notes Report Pack [link].  To steal a few lines from their data sheet:

The MessageStats Report Pack for Lotus Notes extends MessageStats by gathering Domino server configuration details and Notes mailbox content information from a Lotus Notes environment. This report pack can be used as an assessment tool when planning a Notes migration to Exchange or Exchange Online (BPOS), and as an analysis tool for a mixed Exchange and Lotus Notes environment. MessageStats provides detailed reporting on mailbox sizes, security, and delegates, as well as reports about mailbox content.

NOTE:  Don’t confuse this with the Message Stats add-on that used to come with Notes Migrator for Exchange.  That tool simply reported off of the data collected by the migration tool.  This is a completely new tool, written completely from scratch, that reads information directly from your Notes/Domino environment.  It is so new in fact that it is not mentioned on many of the Message Stats overview page yet, but you can find the complete data sheet in the Message Stats library [link].

Meanwhile, these screen shots give you the idea of the types of things the product can help you with:

EncyptMessages

InactiveMbx

InventoryMbxUsers

InventoryServers

LinkedApps

Links

MbxDelegates

MbxSecurity

MbxSizes

OrphanedMbx

PrivFolders

SharedFolders

StorageDetails

StorageServers

When to elevate

If you are running on Windows machines with User Account Control turned on, you may have noticed problems running the Notes client and Notes Migrator for SharePoint – or perhaps even two Notes Migrator for SharePoint clients – at the same time.  

There are two keys to understanding what is going on here:

  1. Notes clients do not allow two Windows accounts to log in or access local Notes resources at the same time.
  2. If you run a process with elevated privileges (aka “Run As Administrator”, aka “Do you want to allow this program to run…?”) in Windows, the Notes client sees this as a different user than if you ran it without elevated privileges.

Therefore you cannot have two processes that access the same local Notes resources unless they are ALL running as Administrator or NONE of them are running as Administrator.  This includes the Notes client itself, the Notes Migrator for SharePoint Migration Console, the Notes Migrator for SharePoint Designer client, and any other Notes add-on products.

This problem gets worse when you realize that in Windows 7 and Windows 2008 usually want to  run Microsoft Management Console programs (including NMSP Migration Console) with elevated privileges (at least on my machine – there may be a way to turn this off).  This pretty much forces you to run Notes and the NMSP Designer as elevated too.

As always, be aware that if the local Notes state get corrupted, then NOTHING will work until you kill all Notes processes, including the ones in the background.  This can make debugging your Notes issues a lot more confusing.  Running Kill Notes [link] is usually a good way to make sure you are working from a good starting point.  (Oh yeah, you may have to run that as administrator too.)

Advice for teaching yourself Notes Migrator for SharePoint

Of course, the entire product is documented in our extensive user manual.  Since not everyone is inclined read such a manual end-to-end, and not everyone is able to take our official partner training courses, I am often asked to suggest other ways to learn how to get started with Notes Migrator for SharePoint and how to accomplish specific tasks.

First and foremost, install the product.  You can get a free evaluation version here or join the 6.0 beta program here.

To learn the basics of using Notes Migrator for SharePoint:

For a drill-down into advanced subjects, as needed:

Generating Microsoft Word Documents

image

This feature allows you to generate Microsoft Word 2007 / 2010 (OpenXML) documents from Lotus Notes documents. Microsoft Word is a very flexible and powerful environment and NMSP supports a wide range of migration scenarios:

1. Migrate Notes rich text to new (blank) Word document – In this scenario the user wants to map the rich text portion of their Notes documents (typically the “Body” field) to Word 2007 / 2010 documents (DOCX files) created from scratch. The generated rich text content is pretty much the only thing in the resulting documents. The fidelity of the migrated content is very high and includes tables, fonts, bullets, images, doc links, and much more. The known limitations are listed at the end of this document.

Optionally, the user may want to migrate specific Notes data items to Standard Document Properties such as Author, Category or Description within the generated Word documents.  As these documents are generated, they will be checked into the designated SharePoint library.

A large number of existing Notes Migrator for SharePoint features apply here, including the ability to set created/modified metadata, document permissions, target folders, content types, workflow state, version history, data transformations and more.

Furthermore, the user may want to migrate specific Notes data items to specific metadata columns in the document library. When the document is opened in SharePoint, these properties will appear as “Server Properties” in the Word user interface.

2. Migrate Notes rich text to new Word documents based on existing templates – This scenario is similar to the one above except that the user can specify the Word 2007 / 2010 template (DOTX or DOTMX file) they want to start with. The generated documents will pick up the headers, footers, backgrounds, etc., of the original template. If the template includes any rich text content, the migrated content will be appended to it.

If the Word template defines Custom Document Properties, the user may want to migrate specific Notes data items to these Custom Properties as well as to the Standard Properties described above.

3. Migrate Notes documents to Word documents with embedded content controls and legacy controls – In this scenario, the user is using the Word template as a “form” with rich layout and data entry fields (and possibly data validation rules, computations, actions, etc.). For each Notes document, Notes Migrator for SharePoint will generate an equivalent Word document with specific Notes data items mapped to these controls.
 
There are several distinct cases here:

3a. If the user is taking advantage of SharePoint’s automatic mapping of SharePoint meta-data columns to Word content controls (exposed to SharePoint developers as “Quick Parts”) then the best approach is to simply map the Notes data items to the metadata columns in the document library and let SharePoint synchronize them with Word.

3b. Notes Migrator for SharePoint can also set the contents of the content controls or legacy controls directly with the mapped Notes data. They will continue to appear as data entry fields in the document, but they will not be synchronized with SharePoint metadata columns.

3c. Notes Migrator for SharePoint can also replace the content controls or legacy controls entirely with the mapped Notes data. They will no longer appear as data entry fields.

Note that in each of the above three cases, the user may still want to map a Notes rich text “Body” field to the document in addition to populating the controls, as described in the first two scenarios.

 

Here are three samples of Notes documents from our standard demo database that I converted to Word:

No Template: [White Oak.docx]

On Quest Letterhead: [Eastern White Pine.docx]

On Data Sheet Template: [Swamp White Oak.docx]

 

Using This Feature

When designing Target Data Definitions in NMSP, you can now add target columns of type WordDocument. You must also specify a column Name (which is used in the field mapping process).

image

Specifying a Template is optional. If you do not specify one, you will always get plain word documents containing whatever rich text contents and standard properties you choose to map from Notes. If you want to specify a template, simply click on the Template property and press the details button to launch the Word Template Options dialog.

image

In the Word Template Options dialog you can optionally import a Microsoft Word 2007 / 2010 template (.DOTX file) using the Import button. Note that a complete copy of the imported template will be saved as part of your Target Data Definition and will be used as the basis for any documents that you generate with it.

Word Template Options dialog also includes a complete list of the available Mappable Fields, which are the parts of the generated Word documents that you might want to map Notes data to. This list includes the main rich text Body of the document, all the Standard Properties available in every Word document (Author, Created date, Subject, Title, Keywords, Category, Status and Revision) and well as any Custom Properties that may have been defined by the Word template you loaded.

If desired, you can customize how these Mappable Fields appear on the Mapping tab. (Recall that Notes Migrator for Sharepoint maintains the distinction between the reusable Target Data Definitions that describe the schema of your SharePoint targets and the mapping of source columns to target fields in a specific migration job.) You can uncheck certain Mappable Fields that you do not want to show on the Mapping tab. You can also override some of the Mappable Field properties such as the MappableName that is visible on the Mapping tab, the AutomapNames property that provides a hint as to which Notes source columns should automatically map to the target field, and the AllowMultiple property which controls when mapping of two or more source fields to one target field should be allowed.

Note that your Target Data Definition may well contain additional target fields that are not part of the generated Word Document. In particular, you can add target columns for any additional metadata properties that should be written to the SharePoint document library, rather than inside the generated Word documents (as described in scenario #3 above).

You may also want to specify Folder names as well as the alternate locations for embedded attachments and OLE objects that should be migrated separately to SharePoint. These are existing features that are described elsewhere, but they apply equally well to Word documents.

image

One final thing you may wish to enable in your migrated Word documents is the new “Migrate Attachment Icons” feature. While this is not always desirable when migrating to List Items, it looks pretty nice in Word Documents.

image

Once you save your Target Data Definition, the various parts of the WordDocument field you defined will be available on the Mapping tab. Note that a special “Field.Part” notation is used here. In the example above we called the WordDocument field “Doc” so the mappable parts are called “Doc.Body”, “Doc.Title”, “Doc.Status” and so on. There is also a special field called “Doc.FileName” that allows you to set the names of the generated Word files from dynamic Notes data.

image

In most cases, you will (at a minimum) want to map the Html version of your Notes documents to the Doc.Body field and map the Subject (or a similarly descriptive Notes item) to Doc.FileName. You can also add additional mappings for standard properties and custom properties as needed.

When you run the job, Notes Migrator for SharePoint will generate one Word document for each Notes document. You can inspect the migration log for any issues.

 

Take Command of Your Notes Application Migration to SharePoint 2010

Here is a webcast I recorded in December that covers new SharePoint 2010 capabilities (and how to migrate to them from Lotus Notes).  Along the way, it shows off a number of new Notes Migrator for SharePoint 6.0 features:  http://www.quest.com/events/ListDetails.aspx?ContentID=13273

PS:  In case you missed it, the beta program Notes Migrator for SharePoint 6.0 is still running.  Click here for details.

Migrating Notes workflows that implement “dynamic” document security

We have discussed various issues about workflow in the past [link].  To make a long story short, there is no way to press a button and magically translate Notes code-based workflows to SharePoint declarative workflows, but Notes Migrator for SharePoint does an excellent job at migrating your workflow state. 

Recently we have been getting a few questions about how people should handle workflows that automatically change the security of the document as it moves through various workflow stages.  For example, imagine a Notes application that lets an “Author” add or update travel requests until they are ready to submit it.  Once the request is submitted, however, the Author can no longer make changes and it us up to her “Manager” to approve it, reject it, or make further changes.  Real-world Notes applications can easily have many such dynamic access rules.

The way Notes developers typically implemented this type of workflow was was to have buttons, formulas and agents that implemented the right steps.  The actual security rules were usually implemented using reader / writer fields that listed specific people (or groups) that were currently allowed to update the document.  So when the Author pressed the “Submit to Manager” button, some code would run which looked up the manager and updated the reader / writer fields on that document so that now only the manager could edit it.

(Side note, there are also some cases where people did the security enforcement in the user interface.  That is to say, they tried to hide the parts people were not allowed to see in the UI.  But most Notes developers understood that this was not true security.)

So when we migrate this document, we have the option to map those reader / writer fields over to the equivalent document-level permissions in SharePoint.  Notes Migrator for SharePoint makes this trivial.  But it is important to remember that the security on this document is really just a snapshot of its current workflow state.  A workflow process on the SharePoint side is going to need to pick up the document from there. 

In the SharePoint world, you will really have two choices.   If you are using something like InfoPath to design complex forms, it will be possible (though not necessarily a good idea) to implement the workflow the old way:  with code on your form buttons and other form events.  It is possible that you could write code that set the appropriate document level permissions as described above. 

It is more likely however that you will want to switch to declarative workflow instead.  Declarative workflow means that you have a workflow engine interpreting a set up rules that someone (not necessarily a developer) has configured.  This could be a built-in SharePoint workflow, something designed in SharePoint Designer, or something designed in Visual Studio.  Another great choice is to implement the workflow with an add-in tool such as Nintex.  Regardless, your workflow definition will include rules to update the document level permissions according to your business rules.

The workflow capabilities in SharePoint Designer get stronger with each new release.  For example, SharePoint 2010 now has explicit actions for Adding, Removing, or Replacing the permissions on a list/library Item.  This is one more example of something powerful you can do in SharePoint 2010 without writing code!

image

Note that you will not see these particular actions in the standard drop-down action list; you will only see them if you are inserting actions into an “Impersonation Step”.

Cool stuff in SharePoint 2010 that Notes customers will love: Declarative Workflow

When people think of complex custom Notes applications they think of workflow.  Workflow is an especially important consideration when migrating Notes applications because migration tools, which can do a great job migrating schema, content, security and forms, have a hard time translating Notes workflows to SharePoint workflows.

Notes workflows are almost always implemented as code attached to various buttons, form events, and agents.  By contrast, the Microsoft platform encourages you to use declarative workflows where you express your workflow as a set of rules that can be entered, modified, and (best of all) understood by a non-programmer.  You can write code if you need to, but this code is usually confined to “activities”, the units of action that you wire together in your declarative workflow.

Declarative workflow capabilities have improved significantly in SharePoint 2010.  First of all, you get a much richer set of out-of-the-box workflows that you can use in your applications.  This list depends on what edition of SharePoint you install and which SharePoint template you are using, but at a minimum it should include an Approval Process, a Collect Feedback review process, a Collect Signatures review process, and a Disposition Approval workflow for managing content lifecycles.  You can now customize the out-of-the-box workflows if needed or design your own from scratch.

image

If you do need to design your own workflows, you can benefit from the new capabilities in SharePoint Designer.  This was a somewhat limited option in SharePoint 2007, forcing many companies to use Visual Studio instead, but those days are gone.  Now SharePoint Designer workflows can address the need of the majority of the workflows found in Notes applications without have to write code.

Users have a much bigger vocabulary of workflow “conditions” and “actions” to choose from, including Notes-like things such as changing the permissions on a document after it is submitted for approval, sending mail notifications, looking up a person’s manager, and moving content to another location.  A particularly powerful feature is to impersonate other users during a workflow so it can perform certain actions using the workflow author’s credentials.  You can even use InfoPath to design nicer forms for collecting information from end users.

image image

You can now design workflows that operate at the site level or the list level and you can reuse workflows across an entire site collection and even export them to use on a completely different SharePoint farm.  Workflows can be visualized as a set of human readable steps and even exported to Visio for graphical display.

The takeaway here is twofold.  First you can replace most of your Notes workflows without writing code.  Save your expensive developers for the hard stuff.  Second, do not attempt to translate all the code that your Notes developers wrote into code on the Microsoft platform.  Instead, devote your energies toward understanding what your old workflows did and make sure you have a deep understanding of how to accomplish these same things in a declarative workflow world.

Migrating workflow state and approval processes

As described previously, when migrating Notes workflows to SharePoint workflows you are really switching from code-based workflows (logic encoded in button events, agents, etc.) to declarative workflows.  And using Notes Migrator for SharePoint, you can migrate your workflow state so you can preserve in-progress workflows between the two systems. 

To achieve this, simply migrate the  (status fields, next review, due date, whatever) as data columns in your list items, InfoPath documents, Word documents, etc.  Depending on how you designed your workflow, you may be done at that point, or you may need to customize it.  The relevant part is that when the workflow starts up for newly written list items,  it should be designed to examine these columns and “advance” to the right state.  For example, the out-of-box “three state workflow” does this but others may not. 

A special (but very important case) is when you are able to replace your “approval process” workflows with SharePoint’s built-in approval process.  This can be a huge win for reducing development costs and eliminating the need to maintain the application later.  What used to be a custom developed application can be replaced with a checkbox feature! 

image

Of course, Notes Migrator for SharePoint correctly migrates the state for that as well.  The relevant part of our documentation can be easy to miss, so I will repeat it here:

Advanced: Customizing SharePoint Data Target Definitions

The Type property can be set to any of the available field types:

• Approval Code – Sets the approval status of documents that are migrated to SharePoint. Input data must be one of the following strings: “Approved”, “Denied”, “Draft”, “Pending”, “Scheduled”.

In other words, map some piece of data (probably using a formula) that yields the values “Approved”, “Denied”, “Draft”, “Pending” or “Scheduled” to a Target field of type “Approval Code”.

There is a lab on this on our support site:  https://support.quest.com/SUPPORT/index?page=solution&id=SOL52223  (Lab 4.5).

Selection Formula madness

Notes Migrator for SharePoint includes the ability to use the Notes formula language for selecting documents. 

image

This is almost exactly the same thing as specifying a Selection Formula for a view in Domino Designer.  I say “almost”, because you can use things like the current time of day or the current user, which you can’t do in Notes.

Of course not everyone running migration jobs is a Domino developer.  So we get a lot of questions along the lines of “how do I write a formula that does such and such?” 

Here are is a listing of some of the most commonly requested examples:

Select documents that were created with the “Memo”form:

FORM = “Memo”

Select documents that have attachments:

@Attachments > 0

Select documents that are under a certain size:

@DocLength > 100

Select documents based on certain keyword values:

@IsMember(status; ‘open’:'delayed’)

Select documents that were created after a certain date:

@Created > [12/31/2006]

Select documents that were last modified after a certain date:

@Modified > [12/31/2006]

Select one document based on it’s UNID:

@Compare(@Text(@DocumentUniqueID); “123456789012345678901234567890ab”; [CASEINSENSITIVE]) == 0

Exclude one document based on it’s UNID:

@Compare(@Text(@DocumentUniqueID); “123456789012345678901234567890ab”; [CASEINSENSITIVE]) != 0

Select response documents only:

@IsResponseDoc

Of course you can use AND and OR to combine expressions, etc. 

Remember that Notes views can have selection formulas too.  If you find a view that has the filtering you need already built into it, you can get much better performance by using that view because the select formulas may have been evaluated already and the results cached on your Domino server.

Specifying a view and a formula together can be very powerful as well.  In this case, the formula is only evaluated on the documents selected by the view and a strict subset of the view contents is selected.

Follow

Get every new post delivered to your Inbox.