The short version: If you have installed the Service Pack 1 that is now available for Dynamics CRM 2016 do not uninstall it.
Not even if Microsoft asks you so they can provide you with a hot fix to fix the bug that is affecting the email router and the server side sync which is causing the systems to fail to create email activities from POP3 email servers.
If you uninstall the service pack first you immediately get a 404 error and you are not able to access crm anymore.
If you have taken a full image of your crm server and you roll it back crm will start working but if you look in the trace files the system will have become completely unstable. Workflows are now failing, several features like exporting data as excel are failing and you will notice that the graphics in the user interface are showing a pretty funky look.
If you were unfortunate enough like we were and have done this to your production environment that is hosting organizations for multiple customers your support team is now getting multiple emails from your customers reporting the failing features. You need to quickly recover the environment.
After many hours of researching the problem so we could find out if we would be able to some how correct it this is what we have found:
- When you rollback the service pack 1 by going to Control Panel, select Programs>Uninstall a program in the crm server the service pack will be removed from the application server. If you have a deployment with the crm back end role deployed on a separate server you would also have to remove the service pack there and also in the sql server to remove the report extensions component.
- After you complete these operations we found out that the software components seem to have been removed. We say “seem” because we really didn’t have a way or time to accurately confirm the software components had been completely rolled back like for example by analyzing the version of the different files and assemblies.
- The upgrades done to the databases of the organizations during the installation of the service pack are not rolled back.
- The most important clue we got of what might have happened and gave us some direction on where to dig in order to try to find a solution happened when we opened the deployment manager. The build version showing for all the organizations was 18.104.22.168 which is the version of the crm 2015 update 0.2.
The first thing we tried to do was to run a repair of the service pack. That didn’t change anything. What was failing before was still failing after the repair.
Another fix we tried was to do an upgrade of the organizations but it failed with several errors including that the version wasn’t correct for the upgrade to the build version 8.1.0.0359 which is the service pack version. This was another clued that something was wrong with the versions.
Before trying anything more drastic we restored the database of one of organizations to a crm 2015 in our development environment containing crm 2015 update 0.2 and tried to import it. A total failure here.
Basically at this point we realized that the databases were completely unusable unless we were able to pin point what was wrong with the versions we would not be able to fix crm.
From this point on we started focusing on finding out what was wrong with the build versions. To start with we realized that removing the service pack 1 had gone pretty wrong because the organizations were showing as 22.214.171.124 which is the version of crm 2015 update 0.2. Even though that was the case we were not able to import the organization into crm 2015 with the update 0.2 installed so something else had to be wrong.
We started by getting a hold of all the build numbers. You can find them here.
Next we used some TSQL scripts that would allow us to inspect all versioning at the database level. We used the following 3 scripts:
FROM [MSCRM_CONFIG].[dbo].[ConfigurationMetadata] order by installedon desc
select revision, majorversion, minorversion from buildversion
FROM [<your database name>].[dbo].[UpgradeActionTracker]
order by UpgradeToVersionNumber desc
If you look at the results of these scripts below you can see that the databases were all showing that they were updated up to service pack 1 which build version is 126.96.36.1999 except the build version of the org database.
Further more if you list all the fields of the BuildVersion table you can see that most of the fields have the version of the the service pack 1.
With the versions in the database this messed up it was not a surprise that we could not import the databases anywhere or do any organization upgrade. The only way it seemed to us we would be able to fix crm would be to find out exactly what these numbers should be and manually correct them.
The first thing we did was to build a crm 2016 in our development environment and create a test environment.
Next we ran the scripts above so we could see exactly what the version numbers should be in the 3 tables, ConfigurationMetadata, BuildVersion and UpgradeToVersionNumber.
We then updated crm with update 0.1 and took screen shots of the results of the scripts and finally upgraded crm with service pack 1.
When we ran the scripts and compared the results from the test organization with the results from one of the damaged organizations we confirmed that everything was the same except for a few fields in the BuildVersion database.
We then manually edited the BuildVersion of the damaged database and corrected the fields that were different. The fields that had to be corrected in the BuildVersion table are the following:
- BuildNumber -> 0
- MajorVersion -> 8
- MinorVersion -> 1
- Revision -> 359
After making these corrections we logged in to crm and now the graphics in the pages were normal, workflows were firing without errors and we could export views to Excel without any errors.
One more thing was left to do. The Deployment Manager was still showing the incorrect version. We disabled, deleted and re-imported each of the organizations in the Deployment Manager.
The import of the organizations went through without any problems and all organizations were now showing the correct build number.
Hope this helps anyone else running into a similar problem.