Notes SharePoint Blog

Steve Walch's blog about his favorite migration tool and other things related to Lotus Notes migration projects

Category Archives: Formulas

Introducing the Notes Migrator for SharePoint 6.1 beta program

Notes Migrator for SharePoint 6.1 is a significant product release that pushes the product further in three important areas:  Design Migration, Content Migration and Pre-Migration Analysis. 

Design Migration

  • Migrate Notes Views to SharePoint Views
  • Content Type Generation
  • New Column Provisioning Options
  • Provision Calculated columns
  • Provision Lookup Fields

Content Migration

  • Migrate by Form Used
  • Improved Content Type Support in Migration Jobs
  • Normalize multi-valued items to multiple documents
  • Dynamic link tracking for URL columns
  • New Run Job options
  • Provision Alternate Libraries

Analysis / Migration Console

  • Capture Design Copies during Analysis
  • Classify by Last Used (All Replicas)
  • Import data into repository from CSV files

General

  • Performance and scalability for Migration Console
  • New User/Group Mapping options
  • Improved CBA/FBA support
  • Windows Authentication using alternate account for Link Tracking
  • Bulk Editing of certain database / class / job properties
  • Usability improvements

 

Once again, the beta will be managed on the new SharePoint for All community site.  If you would like to participate in the beta program, go to http://communities.quest.com/groups/notes-migration-product-beta-group.  Sign in with your Quest Community ID, or register to create a new one.  Then press the “Ask To Join This Group” button.

image

One of the site owners will review your request and will typically approve it the same day.  You will receive a notification and then get full access to technical content and (of course) the beta build itself.

Quest Support will not be able to help you with this version until it releases, so please use the group’s Discussion area for any questions, problems or suggestions.

Selection Formula madness

Notes Migrator for SharePoint includes the ability to use the Notes formula language for selecting documents. 

image

This is almost exactly the same thing as specifying a Selection Formula for a view in Domino Designer.  I say “almost”, because you can use things like the current time of day or the current user, which you can’t do in Notes.

Of course not everyone running migration jobs is a Domino developer.  So we get a lot of questions along the lines of “how do I write a formula that does such and such?” 

Here are is a listing of some of the most commonly requested examples:

Select documents that were created with the “Memo”form:

FORM = “Memo”

Select documents that have attachments:

@Attachments > 0

Select documents that are under a certain size:

@DocLength > 100

Select documents based on certain keyword values:

@IsMember(status; ‘open’:'delayed’)

Select documents that were created after a certain date:

@Created > [12/31/2006]

Select documents that were last modified after a certain date:

@Modified > [12/31/2006]

Select one document based on it’s UNID:

@Compare(@Text(@DocumentUniqueID); “123456789012345678901234567890ab”; [CASEINSENSITIVE]) == 0

Exclude one document based on it’s UNID:

@Compare(@Text(@DocumentUniqueID); “123456789012345678901234567890ab”; [CASEINSENSITIVE]) != 0

Select response documents only:

@IsResponseDoc

Of course you can use AND and OR to combine expressions, etc. 

Remember that Notes views can have selection formulas too.  If you find a view that has the filtering you need already built into it, you can get much better performance by using that view because the select formulas may have been evaluated already and the results cached on your Domino server.

Specifying a view and a formula together can be very powerful as well.  In this case, the formula is only evaluated on the documents selected by the view and a strict subset of the view contents is selected.

CBTs available for Notes Migrator for SharePoint

Our EMEA support team (one of three global support centers where you can get awesome support on my product) has recently put together a series of Computer Based Training labs on installing and using Notes Migrator for SharePoint.  Of course their immediate objective was to better support customers who might be struggling with certain aspects of the product, but (as usual) these guys have over-achieved and what they delivered was an entire training course! 

The table of contents for the first release is shown below.  Each of the links points to a “solution” on the Quest SupportLink web site.  A login is required.  Any Quest customer should have an account on this site, but if you don’t have one, a really good way to get one is to register to download a free trial version of Notes Migrator for SharePoint [link].  This automatically gives you 30 days of free support, including access to the SupportLink site where you will find a wealth of information about all Quest products. You can download the entire set of labs packaged into a single downloadable installable file (48mb) here https://support.quest.com/SUPPORT/index?page=solution&id=SOL52223 or use the individual links below.

Thanks to Maxim and Nerio on the EMEA Migration/SharePoint Support team for all your work on this and your ongoing support of our customers.  I think you have proven yet again that Quest support is not your average call center!

 

1.Basic configuration of Notes Migrator for SharePoint components

• Lab1. NMSP Console Configuration
• Lab2. Configuring NMSP Services
• Lab3. Configuring Global Options using Console (Designer)

2. Using NMSP Console

• Lab1. Discovering migration sources
• Lab2. Analyzing migration sources
• Lab3. Managing migration sources
• Lab4. Migrating data to SharePoint
• Lab5. Automatic target/job assignment
• Lab6. Reports and custom reports
• Lab7. Views and custom views
• Lab8. Managing Tasks

3. Document library migration using NMSP Designer

• Lab1. Standard document library migration to a SharePoint document library
• Lab2. Standard document library migration to a custom SharePoint list
• Lab3. Standard document library migration to a SharePoint Form library
• Lab4. Standard document library migration to ASPX (Basic) pages
• Lab5. Standard document library migration to HTML pages

4. Extending migration

• Lab1. Keywords migration
• Lab2. Migration of multi-valued keywords (e.g. Categories)
• Lab3. Working with strings using Notes formula language
• Lab4. Customizing migration path using Notes formula language
• Lab5. Migrating approval state

5. Permissions migration

• Lab1. Configuring user mapping via ADSI/LDAP lookup
• Lab2. Configuring user mapping via Text File lookup
• Lab3. Configuring user mapping via Domino lookup
• Lab4. Configuring user mapping via Notes Database lookup

6. Quick Place migration

•  Lab1. Migrating Quick Place

7. Domino.Doc migration

• Lab1. Migrating Domino.Doc

 

More labs are coming.  Please send us your suggestions.

Formulas that deal with multi-valued items or multi-valued results

Everyone knows that Notes Migrator for SharePoint has the ability to run formulas (using the Lotus @Formula language) to transform data as you migrate it.  This feature is used for a variety of things ranging from concatenating two fields to translating status code field values to using an @DBLOOKUP to “join” data from a different view.  People are sometimes unclear, however, how they would use this facility to process multi-valued data.

There are two parts to this problem:

1.  How to create a formula that yields multi-values items:   

There are many constructs in the Notes formula language that give multi-valued items as results.  The simplest is to just use a colon “:” to concatenate text values into a text array.  Two examples are shown below:

    ‘A’ : ‘B’

    @Uppercase(Region) : @Lowercase(Region)

Also note that many @Formulas will yield a multi-valued result if the input was multi-valued to begin with.  For example, if Categories is a multi-valued item, then this formula will yield a multi-valued result:

    @Uppercase(Categories)

Be sure to specify the MULTI option in your Source Data Definition column so that the query engine extracts multi-valued data.

image

2.  How to map multi-valued columns to SharePoint fields:  

Only three field types in SharePoint allow multiple values (Choice, Lookup and User).  For these, you need to specify MultiValue in the appropriate Target Data Definition field for NMSP to attempt to write data that way.  (If you are writing to an existing target, you may need to modify the column properties in List Settings as well.) 

image

For other SharePoint field types (such as Text items) you have to make a choice as to whether you want to concatenate the multiple values into a single value, take just the first value, or take the last value.  You can control this option via your detailed mapping options.

image

Migrating Notes keyword fields to SharePoint

Lotus Notes applications have a concept of “Keyword” fields which are basically text fields with a predefined set of possible legal values.  In a data entry form, these may appear as combo boxes, list boxes, check boxes, radio buttons, or pop-up dialogs.  In some cases, Notes developers will even write their own user interface for such fields.  In the following screen shots, the DocType field is implemented as a simple Dialog List.

image

In Domino Designer, we can see how the data entry field is declared.  In this case, it is a single-valued field and the list of possible values is just a list of hard-coded strings. 

image  image

Before going any further, it is important to point out that there is really no such thing as a “Dialog List” data type in the back-end Notes database (and you won’t see any such concept when using Notes Migrator for SharePoint to migrate the content).  If you look at Document Properties, you can see that DocType is really just a Text item.

image

So in Notes Migrator for SharePoint, you could extract this as you would any other text item.  The following screen shots show selecting the database, selecting the data item, and previewing the results.

image  image

image  image

On the SharePoint side, you could choose to map this field to either a “plain” Text column or to a Choice column in your SharePoint List or Document Library.  In particular, if you use our Load From Source Fields function, you will get a simple Text column:

image 

Choice columns.  If you wanted to target a Choice column instead, simply change the Type property to “Choice”.  If the SharePoint list/library you are writing to already has the Choice column defined, it may be programmed with a set of “legal” choices.  In some cases you may want to stick to the list that is there (and even reject records that don’t match the approved list).  You do, however, have the option of using the actual Notes data to automatically extend the list of possible choices on the SharePoint side.  To do this, simply set the AddMissingChoices property to “True”.

image  image

As a reminder, Notes Migrator for SharePoint can actually provision the entire list/library on SharePoint, or it can upgrade the schema of existing lists/libraries by adding additional fields.  But the tool will not convert existing Text columns, etc., to Choice columns.  If you already have a Text column that you want to become a Choice column, you will have to change it manually, or remove the column in SharePoint and let Notes Migrator for SharePoint recreate it the right way.  The following screen shot shows a Choice column provisioned by our tool (with AddMissingChoices enabled).

image

That pretty much covers how to migrate a simple keyword field, but there are several important variations on this that you are likely to encounter in real-world Notes applications…

Multi-valued items.  As discussed in other posts, nobody ever accused Notes databases of being good relational databases and multi-valued items are common.  These types of items are stored as text arrays instead of single text values.

image  image

To migrate the multi-valued items be sure to set the Option property to “Multi” in your source data definition and set the MultiValue property to “True” in your target data definition. 

image  image

Finally be sure to set the MultiValueDisposition property on your field mapping to “All”.  (Unfortunately the Default behavior here is to map just the first value, not the complete array of values.  We plan to change that in an upcoming release.) 

image

(For additional details on writing formulas that deal with multi-valued items, see this post.)

Keyword Aliases.  Some Notes applications differentiate between the way data is stored in Notes and the way it is displayed.  This arrangement is useful in cases where the displayed values might change over time or may be displayed differently for different users (in particular, users who speak different languages).  In the following example, the application stores “1″ “2″ and “3″ for the FAQType field even though user sees “Features”, “HowTo” and “Problem”.  

image  image

While one might choose to continue to store values “1″ “2″ and “3″ in SharePoint and use a Lookup column(see below) to enter and display them, it is more common to simply translate them at migration time.  To do this, simply use a Formula field in your source data definition.  Here is an example formula:   @If(FAQType=”1″;”Features”;FAQType=”2″;”How To”;FAQType=”3″;”Problem”;”Other”)

image 

Note that this is just a special case of “data massaging” that you might want to do as you migrate.  Any valid Notes formula will work here.  You can even use an @DBLookup expressions to merge data from other sources.  (And, yes, I know there are more elegant ways to solve the simple problem above.) 

One more thing to point out here is that some Notes developer may have done this work for you and designed a view column that displays “friendly” values using a formula similar to the one above.  In this case, you could just select the View Column (instead of a data Item) in your Notes Migrator for SharePoint source data definition and simply migrate the pre-computed values from the view index.

Lookup columns.  Sometimes the list of possible keyword values are not hard-coded but instead come from other Notes views or even other Notes databases.  

image

Here are two “Use formula for choices” examples: 

@DbColumn(“Notes” : “NoCache”;”" :”" ; “StateCodes”; 1) 

@DbLookup( “”; “”; “ConfigDocs”; “StatusCodes” ; “Choices”)

Without going into too much detail, the first selects a list of possible choices from the first column in a view and the second looks up a particular configuration document that stores the list of possible choices in a multi-valued text item.

When migrating such a field, you have two basic choices.  First, you could migrate it to a Choice column as described above.  This means that you will no longer be using data documents to control the list of possible choices but will be instead storing the choices as part of the SharePoint list/library schema.  This is a simpler configuration, but you have to be a Designer to modify the choices.

Your other option is to migrate to a SharePoint Lookup column.  Unfortunately Notes Migrator for SharePoint cannot provision this type of column for you so you would have to do a little SharePoint design work yourself.  Once you have set things up, Notes Migrator for SharePoint will happily migrate the data for you.  Here are the steps to migrate to a Lookup column.

  1. Create the List that will contain the set of possible choices.  Of course, Notes Migrator for SharePoint would be a great way to create and populate such a list.  (In the above example, you might migrate the contents of the “StateCodes” view to a similar list in SharePoint.)
  2. Create a List or Library that will contain the main migrated documents.  Add a Lookup column that specifies the above List as the data source.  (As explained above, Notes Migrator for SharePoint can not provision new Lookup columns for you.  It may still be useful, however to use Notes Migrator for SharePoint to design and provision the rest of the list for you.  A good trick is to first migrate a handful of documents using all Text or Choice columns and then change the column type to a Lookup column after the fact.)
  3. Now create your Notes Migrator for SharePoint job.  In your target data definition you can specify the “Lookup” in the Type property for the appropriate column.  When you run this job, SharePoint will reject any records that do not contain legal values so be sure to populate all the possible values in advance (step 1 above).  Of course, the tool will log any exceptions that are encountered.

image   image

Follow

Get every new post delivered to your Inbox.