Migrating Workflow Items

​When performing SharePoint migrations, you will inevitably come across lists with associated workflows needing migration.  These can be challenging to migrate – especially if you just looking to migrate the data, and not the associated workflow(s).  Metalogix has a nice write up of how to use their tool to copy workflows (http://www.metalogix.com/help/Migration-Manager-for-SharePoint/SharePoint-Edition/002_HowTo/003_MigrationActions/017_CopyWorkflows.htm), but this article doesn’t help if all you want is the data from a list.  The Migration Manager will not allow you to copy items from a list which are associated with a workflow, unless the workflow and associated content types are migrated as well.I recently ran into this situation during a migration.  We had a list of tasks that were being created by a workflow in our legacy system.  The target system was setup as a clean environment, and while it had the workflow in place, they weren’t setup identically as the source.  The Migration Manager software refused to migrate the tasks unless we migrated the workflow as well.  Our simplest solution was to break the workflow association on the source server, then use the Migration Manager to migrate the list items to the target server.Here’s how we did it: 

• We created a .NET console application (our “MigrationHelper”) to read the source list, grab each field we cared about (including all metadata) and write the data to a temporary list on the same server.  This temporary list could be created/re-created over and over with no ill effects.  It contained all of the data that we needed, but not the associated workflow.

• We used the Migration Manager to migrate the temporary items to the destination list.  Since the list definitions were different, we simply mapped fields from the source to the destination using the built-in mapping functionality of the Migration Manager tool.

• We created an incremental job to migrate changed data in the temporary source list to the destination list.  We had a large number of items in the list and wanted our “go-live” night to require the least amount of data transfer possible.  (I’ll write a future blog entry on how and why you should do this)

The end result was a process that required a small amount of code on our part (reading the source list, deleting/creating the temporary list, copying data to the temporary list) and left the bulk of the migration to the Migration Manager.  Focusing on the limitations of the tool allowed us to work quickly and efficiently.  Yes, we had to write some custom code, but we didn’t have to reinvent the wheel and write code that Metalogix had already written (see my previous blog entry on using the right tool for the job:http://www.timlinenterprises.com/consultingknowhow/Lists/Posts/Post.aspx?ID=5).Have you ever run into a similar problem?  What did you do to solve it?

The Right Tool For The Job

When planning a data migration effort in SharePoint, you will undoubtedly be confronted by the classic build vs. buy decision.  Ultimately, your decision will be based on the changes to the data as it is moved from system to system.  For simple migrations where you are just switching from one provider to another (running the exact same software on both the old and new servers), a simple DB backup of the SharePoint Configuration and Content DBs should suffice.  However, when some type of data transformation is necessary, the decision becomes less clear.

The last project I worked on seemed straightforward enough - a migration from SharePoint 2007 at hosting provider A to SharePoint 2007 at hosting provider B.  However, a few requirements surfaced that made this trickier than a simple DB backup and restore.

The main complexity surfaced when the schema at the new facility was different from the schema at the source facility.  Why?  Because the custom application was modified and many of the fields that used to be used were no longer necessary.  My first inclination was to copy the entire  Content DB, and simply pare down the data to the desired schema.  However, seeing how the new environment was being setup as a completely "clean" environment, this was determined to be an unsatisfactory solution.

The next simplest mechanism for moving the desired data was to export the required lists to Excel spreadsheets, remove the unnecessary columns, and import them at the destination site.  Easy, right?  Not so fast - unfortunately, SharePoint will set the modification and creation information based on the import date, not the dates in the source data.  Since we were dealing with very sensitive data with strict auditing requirements, this was deemed unsatisfactory.

This left us with fewer options.  We could have just written code to copy directly from the source to the destination - but that's never as easy as it first appears.  Many unforeseen problems often arise that can take up days of programming time.  Ultimately, we made the decision to purchase an off the shelf tool to assist in the data migration - "The Metalogix Migration Manager."

Using this tool greatly simplified our data migration - however, it was not without its share of limitations.  Like any tool, it requires enough understanding of the platform in order to work well.  For example, we still needed to understand how the Content Types were mapped; how Workflows were being used and what data was associated with those workflows; how users were assigned to tasks and how to deal with mapping non-existent users in the new environment.

In the end, we ended up writing a fair amount of code to make the migration run smoothly.  One of my colleagues felt that we should have just written the entire data migration by hand.  I disagree.  I feel the investment in the tool was well worth the money and saved us from dealing with the bulk of the data transfer. Instead, it allowed us to focus on some of the more tricky/more interesting aspects of the move.

Using our custom code and the off the shelf tool in conjunction ultimately led to a successful migration.  Could we have done it all on our own and not used the tool?  Sure, but at a significantly higher cost.  Plus, the client ended up with a license for a tool that could be used for future migrations.

Bottom line - if you are doing any type of data migration in SharePoint, make sure you have someone that understands the platform and consider getting that person the tools they need to be successful.  Trying to save a bit on the tool cost will almost certainly cost you more in the long run.

What do you think?  Have you had a trying experience with a data migration?

Migrating Your Systems - Are You Prepared?

Lately, I've been leading a number of data migration efforts centered around SharePoint 2007/2010 for a few companies with very sensitive data. These companies decided not to upgrade to new versions of software, but decided to change hosting providers. In so doing, they believed that their system migraiton would be a simple process. Not so fast - it's always more complicated than it appears on the surface.


Here's a list of questions to ask yourself when preparing for a system migration: 

• Are we bringing over all of our data, or just a subset?
• What kind of downtime can we accept? 
• Disaster recovery? 
• Which systems get moved first?  Next? 
• Who has access to the old systems? Are those people still around? Will they be part of the migration effort? Do they want to help us? 
• Who will have access to the new systems?
• How are we planning on moving the data? 
• Do we understand our systems and know which systems access which data? 
• Do we know what exact versions of software our current systems are running on? 
• Will our new hosting provider be able to match the versions or do we need to upgrade? 
• How do we ensure the integrity of the data? 
• How much data do we have - how many databases, How many Gigs? How many records? 
• How is the data physically going to be moved? 
• How can we protect the data?  Do we need to audit the entire process?
• Have we inventoried everything at our current provider? 
• Who will be managing the new systems? 
• How much time do we have for the move? 
• What kind of testing will be done to ensure the new systems are working properly? 
• Do we need to hire consultants to help with the move (hint: yes!)

If I could offer just one bit of advice to anyone planning a migration effort, it would be - do not underestimate the amount of effort required.   Hiring an expert who has experience leading migration efforts is well worth the investment.  The cost of hiring a consultant is more than offset by the cost of not having an expert on hand and ending up with major unforeseen problems.  


What do you think? 


Have you recently been part of a system migration? Was it successful?