The Problem
After setting up a new instance of TFS I attempting to use the TFS 2010 Power Tools (Dec ‘11) Team Foundation Backups wizard. However, during the Backup Plan Wizard Readiness Checks, the “Grant Backup Plan Permissions” step failed with the error – Account… failed to create backups using path…
The Fix
Digging into the log created during the Readiness Check I found the following error - Error @xxx Microsoft.SqlServer.Management.Smo.FailedOperationException: Backup failed for Server 'xxx'. ---> Microsoft.SqlServer.Management.Common.ExecutionFailureException: An exception occurred while executing a Transact-SQL statement or batch. ---> System.Data.SqlClient.SqlException: Cannot open backup device 'xxx.bak'. Operating system error 1240(The account is not authorized to log in from this station.).
The implication being that something SQL Server is trying to do isn’t working. To get this fixed I:
- Created a small test DB on the SQL Server box
- Attempted to backup this DB to the TFS backup location – this failed
- Attempted to open the backup location folder from the SQL Server machine – this failed
- Investigated as a possible OS/policy issue
- It turns out that this machine’s security policy - Microsoft network client: Digitally sign communications (always) – was enabled.
- Changed this to disabled and rebooted the server
Voila – I was able to validate and create a new backup plan with the wizard.
The Objective
Three of the 6 development teams using TFS are moving to a different network and domain. There is no on-line connection between the old (source) and new (target) networks.
The objective was for the teams to come in Monday morning, bring up their development machine on the new network and have everything as it had been on the old network.
Failed Approaches
- Clone the data tier and move the data tier to the new network. This failed because the procedures for moving the hardware to a new environment did not work.
- Use the TFS Integration Tools. We had experience with the tools, and after a couple of tries with the 1st approach we decided we wanted a totally new and clean installation of TFS on the new network. However, I found that the tools do not support the migration of test cases and labels. This was unacceptable unless there was no other possible way to do the move.
Successful Approach
I looked into doing a collection move and my initial tests were successful. This approach:
- Allows you to setup a totally new and clean TFS target instance
- Brings much more over than the Integration Tools
Procedure
Don’t attempt this unless you know what you’re doing and have experience with SQL Server and administering all the components of TFS (i.e. SharePoint integration, reporting services, analysis cub processing, source control, work items, etc.…)
The first thing you need to do is install and configure TFS in the new environment. I also suggest moving a small “test” collection first before moving the production collection.
NOTE: My machine names in the source and target environment were identical. This may have helped in some cases simplify the updates to the TFS console in the target environment after attaching the cloned collection.
Be sure to reference the Microsoft procedure as well as the following steps:
- Note down all the team project team members and which security group they belong to
- Apply the latest SPs and updates to all the machines in both networks. Make sure SQL Server is the same version and TFS is the same version in both environments (see TFS console and SQL Server Management Studio)
- Export all your SSRS report definitions (RDL) files. Since we had only 3 team projects to move that used SSRS reporting, I used a PowerShell Script to export the RDL files and then manually uploaded and configured them on the target environment. There is a tool that might be able to automate this, but I was not familiar with it and initial automation attempts were not successful.
- On the source environment:
- Do a full TFS backup
- Stop the warehouse jobs
- Stop the transaction backup job (mine runs every 15 minutes)
- Remove all team project team members from all team project security groups. I’m not sure if this is needed, but I wanted to keep as much data as possible out of the collection that would be invalid in the target environment.
- Detach the collection using the TFS console
- Clone the collection database
- Stop all SQL Server services except the data engine
- Take the collection database offline
- Copy the .MDF and the .LDF files of the collection database
- Bring the collection database back online
- Start all the SQL Server services you stopped
- Attach back the collection (using the TFS console)
- Add back all team project team members to the correct group (using the TFS Power Tools Team Members feature in the Team Explorer makes this really easy)
- Start the warehouse jobs
- Start the transaction backup job
- Verify TFS operations using the TFS Best Practices Analyzer. I also used Grant Holliday’s TFS Administrative Reports Pack to help ensure that cube processing was working correctly.
- On the target environment:
- Copy the database .MDF and .LDF files to the appropriate folders on the database machine
- Attach them to SQL Server to bring the collection database online
- Attach the collection using the TFS console (* see problem below)
- Set the web application on the TFS console SharePoint view to the correct one for the new environment. This will automatically create the collection site for the TFS collection you attached
- Manually create the SharePoint site for each team project in the collection (for team projects that use SharePoint). Create these under the collection site that was created in the prior step.
- Run IISRESET /NOFORCE on the SharePoint machine
- Update the Team Portal settings for each team project from Team Explorer, Team Project Settings
- Make sure all the team project and collection parameters on the TFS console are correct for the new environment
- Update all collection level security groups to ensure members is correct
- All all team members to the team projects security groups
- Rebuild the warehouse from the TFS console
- Import, configure and verify the SSRS reports. Note: I had to configure the data sources for each report even thought the data source name had not changed.
- Process the analysis cube manually
- Setup your backups
- Verify TFS operations using the TFS Best Practices Analyzer and the TFS Administrative Reports Pack reports
Other things I had to do in my specific situation
- Deleted all the build workspaces. This was required to avoid TFS from thinking there was a workspace mapping conflict since the build machines names were the same on the source and target environments
- I remapped the user workspaces to the new owner, changed the computer name portion of the workspace name to match the new name of the users computers (their computer names did change) and provided scripts for them to change the computer name for their workspace.
- I provided scripts for them to replace their shelve-sets to have the correct owner name
- I swapped out the display name in the assigned-to field for all work items with the correct display name (the account IDs were the same in both environments but the display name was not: first last vs. last, first)
- Dropped in a custom assembly we use for setting builds to never be deleted based on build quality
That’s it 
Problems
The only glitch I had was that the collection failed to attach. the error message:
“TF246091: The team project collection cannot be attached because its version ID is higher than the ID for the configuration database. The collection has the following version: Tfs2010.SP1.KB2564498.P#5. The Team Foundation Server is at the following version: TFS2010.SP1.KB2580221.P#4.”
This baffled me at first because from the TFS console and from SQL Server Management studio, the versions of TFS were shown to be the same and the version of SQL Server were shown to be the same.
However, after some digging I found a blog post saying how the TFS version could be inconsistent between the application and the database machine. The resolution was to run:
TFSConfig Update /Reapply on the application machine.
After doing this, the attach worked fine.
I’ve used the platform for ClearQuest migrations to TFS and Subversion migrations to TFS. The setup for testing the configuration can be tricky to ensure you’re starting again from a blank slate.
Problem
I was puzzled after running a number of test migrations to find that no source code would be migrated for some team projects, but not others. I was starting with a new migration configuration, new destination team project and even a re-install of the platform and the SQL Server DB for the platform. For failed migrations the log included the messages
- ConfigurationChangeTracker did not detect any non-transient changes. No cached data will be deleted for session group
- VersionControl: Unable to record sync point for migration source ### of session ### because lastMigratedTargetItem.ItemId is null or empty
For more details on the problem see my forum posting at http://social.msdn.microsoft.com/Forums/en-US/tfsintegration/thread/2717f820-88a3-449b-8074-65dc2e68e23d
Resolution
It turns out that the service account I was using to execute the migration was also used for other TFS services. This account had been added to the root (collection) node of source control by TFS. All the permissions had then been set to deny. I removed the account and TFS added it back, but with no Allow or Deny specified for any of the permissions. The problem was resolved and the migration was able to execute since the account had permissions at the team project level.
The bottom line is that the account being used to attempt the migration did not have needed permissions to the source team project. Once the correct permissions were granted the migration was able to move forward.
We needed to swap out the DB box. Our DB box runs SQL Server, SSRS and SSAS. The AT is on its own box and SharePoint is on its own box.
NOTE: I will update the blog post per resolution of the issue :)
The Microsoft procedure I used was:
- Setup new DB box with WS08 R2 with all updates applied.
- Installed SQL SQL 2008 R2 on new DB box (same version as on the old DB box).
- Took TFS offline.
- Disabled the transaction backup job from existing backup plan (runs every 15 minutes).
- Did a full backup using TFS PT backup wizard.
- Took an image of the AT server.
- Restored to the new DB box using the TFS PT restore wizard. I had to first rename the default report server DBs on the new DB box as they were created by the SQL Server install.
- Change the Database in Reporting Services Configuration Manager, which required that I Redirect Reporting Services to Connect to a Different Server.
- Regarding the connection, the only thing I had to select here was the ReportServer database name
- Regarding the identity update, The correct domain account was already specified and in use. For good measure, I changed it to the Network Service account. I did not get prompted to backup the symmetric key. Then I changed it back to the domain account. I did not get prompted to backup the symmetric key here as well. error: SSRS service wouldn’t start.
- The event error I got was “Report Server Windows Service (MSSQLSERVER) has not been granted access to the catalog content.”
- I had to switch back to the Network Service account and had to manually go and restore the key from the old DB box.
- I then backed up the key. Then changed back to the domain account. This time I did get prompted to create a new key which I did and saved on the new DB box.
- Redirect the Data Source to the Database for Analysis Services
- Prepare SQL Server for Team Foundation Server. This worked fine, but I did open the command prompt using “Run as administrator.”
- Change the Ownership of the Restored Databases. error: –The command“TFSConfig Accounts /ResetOwner /SQLInstance: ServerName /DatabaseName:DatabaseName”failedwith error ”Unable to find any compatible SQL Analysis Services database within the specified instance.” The reason: The analysis DB was not restored! Because it is not backed up by the TFS PT Backup wizard! The fix is to manually backup the TFS_Analysis DB and manually restore it to the new DB box. It doesn’t need to be in-sync because it will be destroyed and rebuilt anyway. Then rerun the TFSConfig Accounts /ResetOwner command. It worked for me.
- I skipped the “Redirect Team Foundation Server to Remote Collection Databases” section because my collection, configuration, SSRS and SSAS are all on the same box.
- Update All Service Accounts.
- The procedure for starting ReportServer and SQL Server Reporting Services is nonsense. Use the “Reporting Services Configuration Manager” to start ReportServer. It was already started for me. Use “SQL Server Configuration Manager” to start SQL Server Reporting Services.
- I used the TFSConfig Accounts /add command for Account Type Application Tier, Reporting Data source and Admin Console (be sure to change the services accounts appropriately.
- Register the Location of the Restored Databases. I was not restoring the application tier and so completed this step. No problem.
- Configure Reporting and Analysis Services. Step 5 in the procedure says “Select the Use Report Server checkbox.” The checkbox label is actually “Use Reports.” error:– in step 22,
- The procedure is wrong when it says “On the report server…” it should say “on the application tier server…”
- Error: the URL to bring up the Warehouse Control Service fails and returns an HTTP 503 error. The URL is: http://localhost:8080/tfs/TeamFoundation/Administration/v3.0/WarehouseControlService.asmx Well, this was pretty frustrating and there was not much on the Internet for this. BUT, it turned out all I needed to do was start the TFS application pools in IIS on the application tier server.The pools are: “Microsoft Team Foundation Server Application Pool” and “…Web Access Application Pool.”
- In the end I was getting access errors to the analysis DB and the Warehouse Control Web Service would simply not return the page when attempting to invoke ProcessWarehouse. So, I reverted back to the old DB box by restoring my AT box. Then I found some lingering issues where the Excel Reports on the SharePoint server had their connection string updated to use the new DB box. It seems TFS is correcting this now. I ran a diagnostic (BPA) and everything came out OK.
Our TFS instance had been humming along for some time when I noticed that the full and incremental cube refresh jobs started failing. Using Grant Holiday’s “Team Foundation Server 2010 Administrative Report Pack” I was able to better visualize and diagnose the problem.
The Problem
The full error message was -
OLE DB error: OLE DB or ODBC error: Login timeout expired; HYT00; A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.; 08001; SQL Server Network Interfaces: Error getting enabled protocols list from registry [xFFFFFFFF]. ; 08001.
Errors in the high-level relational engine. A connection could not be made to the data source with the DataSourceID of 'Tfs_AnalysisDataSource', Name of 'Tfs_AnalysisDataSource'.
The Fix
I had been using the TfsService domain account to run the TFS DB Analysis service, but after looking at John Socha-Leialoha's Blog on Fixing Cube Processing ODBC Errors, I decided to:
- Change the service account for the Analysis Services to Local System
- Manually invoked a full refresh*
That fixed the problem! The full and incremental refresh jobs then started working again. The only thing is that I have no explanation for why the problem suddenly appeared.
* see the post Accentient - Manually Processing the Team Foundation Server 2010 Data Warehouse and Analysis Services Database
If you select a folder in the Source Control Explorer that has an extremely large number of files, the Source Control Explorer may take a long time to load, effectively locking up. If you close Visual Studio and re-open, source control will attempt to open at the same location and the same thing happens. Rebooting will not solve this problem. Here's the solution:
- Open the Task Manager and end the Devenv process to close Visual Studio
- Bring up the VS command prompt from the start menu and run Devenv /ResetSettings. Do NOT open the SC explorer yet
- Go to Tools, Options and select Source Control
- Select Visual Studio Team Foundation Server
- Uncheck the "Open Source Control Explorer to the most recent folder" checkbox.
- Close VS and reopen