Dangling SharePoint Features and Files at Upgrade

I recently upgraded from SharePoint 2010 RC1 to SharePoint 2010 RTM on my development machine.  Being a development machine there were several old projects that had been ripped from SharePoint without deactivating the features.  SharePoint tends to swallow these problems – although they show up in the logs.  Upgrade, on the other hand, is blocked by these problems.

I received several errors at upgrade time including errors like

[OWSTIMER] [SPContentDatabaseSequence] [ERROR] [5/6/2010 1:10:30 PM]: Found a missing feature Id = [11d1f820-10d9-48c9-bc44-4ae4439be635], Name = [FeatureName], Description = [], Install Location = [PackageName_FeatureName]

And

[OWSTIMER] [SPContentDatabaseSequence] [WARNING] [5/6/2010 1:10:31 PM]: File [Features\ PackageName_FeatureName \ModuleName\Files\Custom.css] is referenced [1] times in the database [WSS_Content], but is not installed on the current farm. Please install any feature/solution which contains this file.

[OWSTIMER] [SPContentDatabaseSequence] [WARNING] [5/6/2010 1:10:31 PM]: One or more setup files are referenced in the database [WSS_Content], but are not installed on the current farm. Please install any feature or solution which contains these files.

I was not surprised by the first error.  I had removed features without deactivating them, and how could psconfig know how to upgrade them if they no longer existed in the farm.  The second set of errors was unexpected and spoke to a new layer upgrade I had not expected.  The files that it was complaining about were documents that I have deployed to libraries using SharePoint modules.  The modules were gone, but the libraries existed.  How did it know that the document came from a SharePoint module and why does it care?

Further examination of the dbo.AllDocs table in SQL Server Management Studio turned up that it contains the columns “SetupPath” and “SetupPathVersion” which can tie a document in a library back to a feature deployment action.  Apparently, one of the activities psconfig does during upgrade is to check if the file in the library should be replaced by a newer version that had been deployed into the 14 hive.  I am not sure if that is done by comparing the matching SPPersistedFile within the config database or by comparing the document with the actual file on disk.   The SetupPathVersion is a tinyint so it is hard to understand what it maps against.

Regardless of the algorithm, it is pretty clear what upgrade was attempting to do.  This adds a new item to my list of upgrade functions that psconfig executes:

+ Check SPPersistedObject versions in the config database and call the IUpgradable and IUpgradable2 interfaces to get them to upgrade as needed.

+ Traverse the SPSite-SPWeb trees in the farm and upgrade each feature instance using the feature upgrade CAML along with any SPFeatureReceiver.FeatureUpgrading methods.

+ Examine all Library documents, determining if any come from features.  If they do come from features, check to see if a new version needs to be loaded from the 14 hive.

In my situation the upgrade failed and directed me to fix errors (shown in log) and try again. The error may say: “Please install any feature or solution which contains these files.” I thought this would be rather risky with my half upgraded farm. To complete my upgrade, I went to SQL Management Studio and deleted the problematic data from my content database.  I queried the dbo.Features table based on the feature IDs in the log and deleted those rows.  I queried the dbo.AllDocs table based on the SetupPath of files found in the log and deleted those rows.  I kicked off:

PS> psconfig -cmd upgrade -inplace b2b

Which completed successfully and I was upgraded.

 

Features are Versioned and Upgradable on SharePoint 2010

One of the big sticking points with SharePoint 2007 was that Microsoft failed to adequately address was how to update between versions of third party SharePoint applications. Microsoft has addressed much of this introduction of feature versioning in 2010.

Starting with SharePoint 2010, the versions of all features activated are stored. Farm and web application features versions are stored in the config database. Site collection (SPSite) and site (SPWeb) features versions are stored in the content database. Features shipped without version numbers default to version 0.0.0.0.

With the in-place upgrade process provided in psconfig.exe, the farm is scanned and the versions of all features activated are compared with the versions of the feature definitions installed. Any active feature that is not up to the current feature definition (SPFeatureDefinition) version has the opportunity to upgrade. How that upgrade occurs depends on a new block of CAML XML in the feature definition under the new UpgradeActions element.

There are several types of UpgradeActions. They include CustomUpgradeAction, ApplyElementManifests, AddContentTypeField, and MapFile. The most flexible of these is CustomUpgradeAction for which you can assign a SPFeatureReceiver. When you do that it will invoke a new method FeatureUpgrading on the SPFeatureReceiver.

Some upgrade code paths must always run. An example of this would be having a base schema for a content type which never changes (because it was released in the past), but always AddContentTypeField regardless of whether deploymenting a new current version or an upgrading to the current version.

At other times, you must use the VersionRange element. This element specifies the affectivity of the upgrade code. An example would be a new module added to older versions.

Over the next few days I will post some examples of using feature upgrade.

Programmatically Activating Features just after Deployment

There are a variety of scenarios where features need to be programmatically activated just after a solution has been deployed.  I have found that even when the SPSolution object says it has been deployed, it really isn’t finished deploying.  Sometimes my SPFeatureDefinitions are either non existents or not installed even when the deployment is “done”.  They eventually do show up. Until they do, feature activation fails when the feature is added to the SPFeatureCollection of the SPFarm, SPWebApplication, SPSite, or SPWeb.  I ended up coding a small bit of code to work around this problem.  This example shows activation of a web application scope feature, but the algorithm can be applied to a feature of any scope.

private void ActivateWebApplicationFeature(Uri webApplicaitonUri, Guid featureId, int timeoutSeconds)
{
    const int sleepTime = 1000;
    for (int i = 0; i < timeoutSeconds; i++)
    {
        try
        {
            SPFeatureDefinition definition = SPFarm.Local.FeatureDefinitions[featureId];
            if (definition.Status == SPObjectStatus.Online)
            {
                break;
            }
            Thread.Sleep(sleepTime);
        }
        catch
        {
            Thread.Sleep(sleepTime);
        }
    }
    SPWebApplication application = SPWebApplication.Lookup(webApplicaitonUri);
    if (application.Features[featureId] == null)
    {
        AddWebApplicationFeature(webApplicaitonUri, featureId);
    }
}

private void AddWebApplicationFeature(string webApplicaitonUri, Guid featureId)
{
    int count = 0;
    while (true)
    {
        try
        {
            SPWebApplication application = SPWebApplication.Lookup(webApplicaitonUri);
            application.Features.Add(featureId);
            application.Update();
            break;
        }
        catch (SPDeletedConcurrencyException)
        {
            // Done. This object is gone so we are finished.
            // Optional logging of condition
            break;
        }
        catch (SPUpdatedConcurrencyException)
        {
            count++;
            if (count > 10)
            {
                throw;
            }
        }
    }
}

The algorithm is to first wait for the feature definition to appear.   When looking up the SPFeatureDefinition in the SPFeatureDefinitionCollection an exception will be thrown until the definition has been added.  The code waits a second between checks.  It continues waiting for the SPFeatureDefinition to install.  Finally it activates the feature if the feature needs activation (in certain upgrade scenarios the feature may already be active).  You will notice that the second method uses my technique to control SPDeletedConcurrencyException and SPUpdatedConcurrencyException. 

Using WindowsIdentity.Impersonate in SharePoint applicatons

Linking legacy code to your SharePoint application can pose several issues.  One of these issues is how to run the code as the correct user.  When threads enter your business logic through IIS and SharePoint they already impersonate the user connecting to the Web Application.  Threads in the SPTimerService run as the timer service process user. Threads in your custom SPWindowService run as the service process user that you have specified.  Getting your thread to assume the right WindowsIdentity can be a challenge.

WindowsIdentity.Impersonate should not be used as a method to access SPSite objects in pure non-legacy SharePoint applications.  That should be done by passing an SPUserToken object into the SPSite constructor.  Still legacy code may require the change in WindowsIdentity and glue code may access the SPSite object to communicate data back into SharePoint.  There are two steps to accomplishing this. 

First convert the SPUser to a WindowsIdentity. To accomplish this, create the user principal name (UPN) for the user.  Then instantiate a new WindowsIdentity using the UPN.

private static string GetUpn(SPUser spUser)
{
    string[] tmp = spUser.LoginName.Split('\\');
    System.DirectoryServices.ActiveDirectory.Domain domain =
        System.DirectoryServices.ActiveDirectory.Domain.GetCurrentDomain();
    return tmp[1] + "@" + domain.Name;
}
.
.
.
SPUser spUser = GetMySPUser();
WindowsIdentity spUserIdentity = new WindowsIdentity(GetUpn(spUser));

Restrictions: This approach will only work when the user is actually a member of the Domain.GetCurrentDomain.  Otherwise you need to search for the domain name through the active directory service. Also this approach will only work if the process user has the local security policy “Act As Part of the Operating System” = “true”.  By default the NT Authority\Network Service user has this privilege. Additionally the WindowsIdentity instantiation is also restricted to the Server 2003 or Sever 2008 operating systems, but so is SharePoint.

Next perform the impersonation.

WindowsImpersonationContext context = spUserIdentity.Impersonate();
try {
    MyLegacyData data = ExecuteLegacyLogic();
    using (SPSite site = new SPSite(mySiteId))
    {
        using (SPWeb web = site.OpenWeb(myWeb)
        {
            AddLegacyDataResults( web, data );
        }
    }
}
finally
{
    context.Undo();
}

Notes: In the above block the SPSite object access is the same as if were performed using new SPSite(mySiteId, spUser.UserToken).

Restrictions: Never run this code block inside a SPSecurity.RunWithElevatedPrivileges delegate.  This appears to be unsupported by SharePoint.  The resulting SPSite and SPWeb objects will be unusable.  In this condition, a StackOverflowExcetion will be thrown if you attempt to access an SPUser in the SPWeb.Users collection and will bring down the entire process.

Outside of the above restriction, the code appears to work as expected.

How to create a new user for an SPWeb

It seems that if a user needs to be added to a SPWeb object one should just call SPWeb.Users.Add. Unfortunately, this does not work and you will get an exception.  To add a user correctly you need to define the users access rights. This is most easily done using the predefined roles.  Here is a small foreign method that does the job correctly.

private void AddUser(SPWeb web, SPRoleType roleType, string login, string email, string name, string notes)
{
    SPRoleDefinition roleDefinition = web.RoleDefinitions.GetByType(roleType);
    SPRoleAssignment roleAssignment = new SPRoleAssignment(login, email, name, notes);
    roleAssignment.RoleDefinitionBindings.Add(roleDefinition);
    web.RoleAssignments.Add(roleAssignment);
}

Remember to always get SPRoleDefinition objects using the SPRoleDefinitionCollection.GetByType method and never using the [string] indexer.  The indexer takes the localized name of the role. Using a string like “Viewer” will work in English but will not in any other language. SPRoleType is an enumeration that is locale independent.