Dangling SharePoint Features and Files at Upgrade

I recently upgraded from SharePoint 2010 RC1 to SharePoint 2010 RTM on my development machine.  Being a development machine there were several old projects that had been ripped from SharePoint without deactivating the features.  SharePoint tends to swallow these problems – although they show up in the logs.  Upgrade, on the other hand, is blocked by these problems.

I received several errors at upgrade time including errors like

[OWSTIMER] [SPContentDatabaseSequence] [ERROR] [5/6/2010 1:10:30 PM]: Found a missing feature Id = [11d1f820-10d9-48c9-bc44-4ae4439be635], Name = [FeatureName], Description = [], Install Location = [PackageName_FeatureName]

And

[OWSTIMER] [SPContentDatabaseSequence] [WARNING] [5/6/2010 1:10:31 PM]: File [Features\ PackageName_FeatureName \ModuleName\Files\Custom.css] is referenced [1] times in the database [WSS_Content], but is not installed on the current farm. Please install any feature/solution which contains this file.

[OWSTIMER] [SPContentDatabaseSequence] [WARNING] [5/6/2010 1:10:31 PM]: One or more setup files are referenced in the database [WSS_Content], but are not installed on the current farm. Please install any feature or solution which contains these files.

I was not surprised by the first error.  I had removed features without deactivating them, and how could psconfig know how to upgrade them if they no longer existed in the farm.  The second set of errors was unexpected and spoke to a new layer upgrade I had not expected.  The files that it was complaining about were documents that I have deployed to libraries using SharePoint modules.  The modules were gone, but the libraries existed.  How did it know that the document came from a SharePoint module and why does it care?

Further examination of the dbo.AllDocs table in SQL Server Management Studio turned up that it contains the columns “SetupPath” and “SetupPathVersion” which can tie a document in a library back to a feature deployment action.  Apparently, one of the activities psconfig does during upgrade is to check if the file in the library should be replaced by a newer version that had been deployed into the 14 hive.  I am not sure if that is done by comparing the matching SPPersistedFile within the config database or by comparing the document with the actual file on disk.   The SetupPathVersion is a tinyint so it is hard to understand what it maps against.

Regardless of the algorithm, it is pretty clear what upgrade was attempting to do.  This adds a new item to my list of upgrade functions that psconfig executes:

+ Check SPPersistedObject versions in the config database and call the IUpgradable and IUpgradable2 interfaces to get them to upgrade as needed.

+ Traverse the SPSite-SPWeb trees in the farm and upgrade each feature instance using the feature upgrade CAML along with any SPFeatureReceiver.FeatureUpgrading methods.

+ Examine all Library documents, determining if any come from features.  If they do come from features, check to see if a new version needs to be loaded from the 14 hive.

In my situation the upgrade failed and directed me to fix errors (shown in log) and try again. The error may say: “Please install any feature or solution which contains these files.” I thought this would be rather risky with my half upgraded farm. To complete my upgrade, I went to SQL Management Studio and deleted the problematic data from my content database.  I queried the dbo.Features table based on the feature IDs in the log and deleted those rows.  I queried the dbo.AllDocs table based on the SetupPath of files found in the log and deleted those rows.  I kicked off:

PS> psconfig -cmd upgrade -inplace b2b

Which completed successfully and I was upgraded.

 

In-place Upgrade of Solutions in SharePoint 2010

The In-Place upgrade mechanism for SharePoint solutions now uses the feature upgrade mechanism to upgrade features already installed and activated in the farm. In this scenario an existing version of a solution exists within a farm with a set of active features. The goal is to upgrade it to a new version without retracting and redeploying. This will allow any active features to upgrade if needed.

Only one version of any solution will exist in the farm solution store. In the first step, a WSP that exists in the solution store is updated with a WSP file on disk. Then a job is run in the timer service and files in the WSP are deployed into the 14 hive on top of the existing files. (This is an update of the SPPersistedFile objects in the config database.) Finally psconfig must be called to upgrade features using the new 2010 feature upgrade system and to upgrade administrative SPPersistedObjects using the IUpgradable and IUpgradable2 interfaces.

The upgrade mechanism extends what was delivered in 2007. Either STSADM or PowerShell can be used to upgrade solutions stored in the farm solution store.

The STSADM approach:

CMD> stsadm –o upgradesolution –name <Solution name> -filename <path to new version of solution> -immediate -allowgacdeployment

CMD> stsadm -o execadmsvcjobs

CMD> psconfig -cmd upgrade -inplace b2b

The PowerShell approach:

PS> Update-SPSolution –Identity <Solution GUID ID or solution name> -LiteralPath <path to new version of solution> –GACDeployment true

PS> psconfig -cmd upgrade -inplace b2b

It is important to note that the “b2b” argument for psconfig. It indicates that the upgrade will action will be executed on a build to build comparison. With “b2b” any feature that has any positive version change will be upgraded. Alternatively the “v2v” argument can be used. The “v2v” argument will cause features to only be upgraded if the major and minor versions increase; it ignores the build and revision portion of the version string.

If the solution to be deployed contains assemblies for the GAC the –allowgacdeployment must be used with stsadm or –GACDeployment true with Update-SPSolution. Watch the time between the completion of stsadm or Update-SPSolution and the completion of psconfig. During this time the GAC will be running the new DLL assemblies while the features will still be configured with the previous version.

The upgrade is global to the farm and all installed components. The psconfig will search the farm for all items that need upgrade and upgrade them in place. This will include all pending SharePoint patches that have been installed as well as any third party solution upgrades that have been place in the system but to which the psconfig –cmd upgrade has not been run.

It is possible to end up with a partial upgrade. All activated features that can be upgraded will. Any that can’t be upgraded will be noted in an upgrade log. The psconfig tool will point you to the log in the event of failure.

I have performed many upgrades on solutions stored in the farm solution store during my testing.

The in-place upgrade for the new SPUserSolution sandbox solutions is very different. It happens automatically if you have set up the WSP file correctly with the same solution ID but a different hash code. Here is what Microsoft says:

An upgrade for a sandboxed solution is any solution that is uploaded to the solution gallery with the same solution ID of the version deployed to the site collection but with a different hash code. Incoming requests fail during the upgrade process. Any feature upgrade actions will be processed as well. During upgrade, the feature definitions for the existing solution are compared with the feature definitions for the new solution. Existing feature definitions are upgraded on the site and all subwebs are removed if they no longer exist in the new solution. All new feature definitions are activated on the site. However, upgrade will fail if an existing feature definition is a newer version than the feature definition that is included in the new solution.

I have yet to try it and see if it all works as they claim.

Best Practices for SharePoint Solutions

Here are my personal requirements that I use for applications built on SharePoint.  I think they embody many of the best practices.

Applications should be SharePoint Solutions

·         Each application should be self contained in a single SharePoint solution .wsp file.

o   Including all DLL’s to GAC

o   Including all SafeControls

o   Including all Templates

o   Including all Features

o   Including all List and Item Schema

·         Including all WebServices  and HttpHandlers for data communication

·         Solutions should be deployable (SharePoint’s concept of an internal install) to a new Web Application or one or more existing Web Applications.

·         Solutions should not be global to the farm (SharePoint has the concept of a global solution deployment) as this prevents multiple versions of the same solution from coexisting on the farm.

·         Solutions must be visible in the Farm Solution Manager so they may be retracted from the SharePoint farm.

·         Solutions should properly clean up themselves upon a Solution retraction.

·         Solutions must play friendly with third party applications

o   Unique ID’s on all objects

o   Unique namespace within file store

o   Never override or modify built in items

o   Smart about web application configuration

·         Solution version to version compatibility must be maintained

o   If the site template needs to be upgraded

§  Never change ID or site template (no delete)

§  Add a new site template into the system with the new functionality

§  Preserve all old template definitions

·         there may be existing site content built on this template

·         there may be customer customized site templates built on this template

·         PTC partners may have built their own templates on top of existing templates

§   Mark old template definitions as hidden so that customers are discouraged from using old site definitions

§  Feature staple new features to old site definitions to add new features to previously installed sites

o   If a list definition needs to be upgraded

§  Never change ID or  remove a list definition (no delete)

·         Never remove a content type

§  add a new list definition into the system with the new functionality

§  Preserve all old list definitions

·         They may have been customized and migration is impossible without loss of customization

§  Mark old list definitions as hidden so that customers are discouraged from using old list definitions

§  Repurpose the Microsoft’s SharePoint Upgrade Schema system to upgrade old lists to new lists

o   If a feature definition needs to be upgraded

§  Never change ID or remove a feature definition (no delete)

§  Do not remove any existing element definitions

o   If a administrative object needs to be upgraded

§  Should maintain deserialization between versions

§  Override the Upgrade method in the SPPersistedObject to perform self upgrade

§  Detect old status (when deserialized) and implement the NeedsUpgrade property

§  Maintain all API’s from version to version (no delete on API)

 

SharePoint uses action states in SPObjectStatus

An action state is a state that indicates that an action is being performed.  The SPObjectStatus used by the SPPersistedObject.Status has three such states Provisioning, Unprovisioning, and Upgrading this allows SPPersistedObjects to undergo long running operations.  Here is a quick summary of all the states:

Name	Meaning	Details
Disabled	OFF	New or fully removed object
Online	ON	Object fully setup
Provisioning	Action State	Moving to ON
Unprovisioning	Action State	Moving to OFF
Upgrading	Action State	Changing object versions
Offline	FAILED	Your dead

One of the major misconceptions with new SharePoint developers is that ON and OFF are really Online and Offline, but they are not. ON and OFF are represented by Online and Disabled.  Offline has the same meaning it does in SQLServer. Your object is dead beyond use.  This is data corruption, hardware failure, network failure, etc.

SharePoint maps the Provisioning and Unprovisioning states differently depending on the object.  Here are some examples:

Object	Provisioning	Unprovisioning
SPWindowsServerInstance	Starting	Stopping
SPSolution	Deploying	Retracting
SPDatabase	Creating	Dropping

It is best practice to code any custom SPPersistedObjects to set the Provisioning and Unprovisioning action states correctly.  Particularly if your Provision and Unprovision methods are long running.

SPPersistedObject.Properties is a great place for storing miscellaneous data

The SPPersistedObject.Properties  property is a great place to store miscellaneous data and place flags on administrative objects.  There are two very important considerations when using this data store.

1. Although it is implemented by a System.Collections.Hashtable which can take any object as a key.  The only acceptable key in SharePoint is a string key.  Using any other object class will cause SharePoint to throw spurious class cast exceptions. The Microsoft.SharePoint.Administration.SPBackwardCompatibilityPropertyMapper.GetClusterProperties() is one such internal method that will throw System.InvalidCastException if your SPFarm.Properties contain non-string keys.

2. Remember to call Update after adding or removing a key/value pair in the Hashtable.  It is easy to miss this and I have found that it is often not detected in unit tests.  The following are SharePoint singleton objects within a given process: SPAdministrationWebApplication.Local, SPDiagnosticService.Local, SPFarm.Local, SPFarm.Local.TimerService, SPServer.Local, SPWebService.ContentService, SPWebService.AdministrationService,  SPWebServiceInstance.LocalAdministration, and SPWebServiceInstance.LocalContent.  In most unit tests both the operations performed and the asserts called occur in the same process.  If you are setting properties on these singleton objects within your unit test process, there is no way to tell whether those properties were persisted back to the database.  The only way to detect it would be to use a two process test harness.

Dealing with SPDeletedConcurrencyException

SPDeletedConcurrencyException is an exception thrown by the SPPersistedObject Update() method.  It indicates that another processes has deleted the SPPersistedObject you are trying to update.

The questions you need to answer are:

·         Is this unexpected behavior for your application (is class not expected to be deleted)?

·         Does the removal of my object negatively impact the current operation?

If the answer to both these questions is “no”, then simply suppress this exception.

Building on yesterdays code snip:

int count = 0;
while (true)
{
    try
    {
        MySPPersistedObject obj = FindMyObject(id);
        obj.SomeOperation();
        obj.Update();
        break;
    }
    catch (SPDeletedConcurrencyException){
        // Done. This object is gone so we are finished.
        // Optional logging of condition
        break;
    }
    catch (SPUpdatedConcurrencyException)
    {
        count++;
        if (count > 10)
        {
            throw;
        }
    }
}

Dealing with SPUpdatedConcurrencyException

SPUpdatedConcurrencyException is an exception thrown by the SPPersistedObject Update() method. It indicates that another processes has updated the SPPersistedObject you have just modified and are trying to update.

I have found that best practice is to wrap all calls to Update on any SPPersistedObject to iterate until successful. Of course it is folly to iterate forever, so I have my code give up after 10 tires. Here is a snip:

int count = 0;
while (true)
{
    try
    {
        MySPPersistedObject obj = FindMyObject(id);
        obj.SomeOperation();
        obj.Update();
        break;
    }
    catch (SPUpdatedConcurrencyException)
    {
        count++;
        if (count > 10)
        {
            throw;
        }
    }
}