20 steps to decommission a redundant data center facility

3 years ago 380

Decommissioning a redundant facility is a harder task than shutting down a standalone operation. Learn the appropriate steps to take to get the job done safely and reliably.

It was with an element of sadness that I recently participated in the decommissioning of a remote site from our global environment. This site had been in operation nearly 14 years, and I helped build it out and maintain it.

SEE: Electronic Data Disposal Policy (TechRepublic Premium)

Since the site was redundant, this meant that we'd be shutting down the physical and virtual servers there as well as the networking gear, but leaving infrastructural elements such as Active Directory, DNS, user and group accounts and so forth in place as those would remain in our other active locations.

This made the site decommission a bit trickier than one I worked on previously; a standalone, single facility that was closing (the business was shutting its doors) and thus it was just a matter of turning out the lights at the end of the month. Here is the outline of the steps we took; note that some will also apply to standalone sites but I wanted to present the entire process from A to Z. This process assumes you have already build a new functioning site and that Active Directory is in place (skip those steps if otherwise).

1. Set the timeline and responsible parties

Document a list of milestones as to what will take place when and factor in the planning process itself. This should outline a reasonable timeframe as to when the following steps should take place and identify who should be involved and what roles and responsibilities they possess.

SEE: Checklist: Server inventory (TechRepublic Premium)

2. Assess inventory

Get a list of every server (physical and virtual) that will be shut down. Add to the list all the network devices: switches, routers, VPN devices, wireless access points and other related elements. Include printers, scanners, badge readers—anything with a serial number. You'll not only need this to determine what's actually going away, but likely it will come in handy when trying to sell, donate or recycle the equipment.

3. Arrange to terminate any third-party vendor contracts

Notify all vendors about what's happening, and when you need service or support cut off. It's probably a good idea to schedule this for a couple of days or a week after the expected shutdown just in case.

4. Ensure critical resources are relocated elsewhere

This is one of the more important steps. In the example of the site I just shut down, the two domain controllers were holding the FSMO roles for Active Directory. And while they were also DNS servers, redundant DNS servers existed in the remaining sites. I transferred those FSMO roles over to servers that would remain intact. We also had application servers in those sites that would take over for the soon-to-be-defunct servers and in fact had built an entirely new site to replace it.

SEE: The future of work: Tools and strategies for the digital workplace (free PDF) (TechRepublic)

5. Plan the shutdown in order

Using your inventory list from step 1, determine what order to shut systems and devices down. Work from low value to critical value—for instance, domain controllers and DNS servers are likely the third to last systems to be powered off, with the second to last ones being any ESX hypervisors followed by the network devices. Don't put yourself in a situation where a high-value device was powered off before a low-value one and now you've lost access to get to something.

6. Notify end users about what to expect

Send out emails explaining what is happening and why, when it will take place and what users should do beforehand and after (if applicable). Include all milestone dates and contact information to which they can direct any questions or concerns.

Now we get to the actual site decommission steps.

7. Remove devices from monitoring

Before you even log in to any of the systems to be shut down, take them out of any monitoring matrix they are in. This will prevent you from receiving and having to deal with a slew of critical alerts that there is a massive outage underway.

8. Remove devices from backups

This is the same as the last step; take the systems you are decommissioning out of any backup rotations. Keep the existing backups until they age out per their existing schedule, however (it might also be wise to keep one archive backup).

9. Shut down non-essential physical servers

Either log in to the servers directly or via any control interface (such as Dell iDRAC) and power off the lower-value servers.

10. Shut down non-essential virtual servers

Either log into the servers directly or via any control interface (such as VMWare vSphere) and power off the lower-value servers.

11. Remove Active Directory and DNS from DCs

At this point your domain controllers should be the last remaining servers standing. Be careful with this step because if you complete this and then find another system you need to access or log in to, you might find yourself unable to do so with these two key infrastructural elements gone.

A word of caution: First check the zone transfers tab on all forward and reverse DNS zones for every domain controller to ensure these servers were not the only ones performing zone transfers to other servers as you'll likely run into name resolution problems down the line.

This step involves launching Server Manager (all Windows server versions since 2008), clicking Manage, choosing Remove Roles and Features, and then proceeding to uncheck the options for Active Directory Domain Services and DNS Server and clicking through the prompts to remove them. You'll be asked if you'd like to Demote This Domain Controller in one of the dialogue boxes and given a link to do so.

There is also a PowerShell method you can research.

If this involves a child domain which is site-specific, when you remove AD from the last domain controller in the site you should click the checkbox on one of the dialogue boxes that this is the Last Domain Controller in the Domain. This will nicely clean things up for you and nuke the domain so the other sites won't see that child domain any longer.

12. Remove obsolete objects from AD Users and Computers

Delete all decommissioned servers from the domain controllers in the remaining site(s) including the defunct domain controllers.

13. Remove obsolete sites/subnets from AD Sites and Services

14. Remove obsolete DNS zones

SEE: Power checklist: Local email server-to-cloud migration (TechRepublic Premium)

15. Shut down the hypervisors

16. Shut down storage arrays

17. Shut down network devices

18. Remove firewall objects and rules

Remove all those pertaining to the defunct site from the remaining live sites (where applicable).

19. Arrange secure disposal of all equipment

Make sure to erase any hard drives.

20. Send a final notification to end users that the work is complete

Set expectations/recommendations for anything they need to know or do going forward.

Data Center Trends Newsletter

DevOps, virtualization, the hybrid cloud, storage, and operational efficiency are just some of the data center topics we'll highlight. Delivered Mondays and Wednesdays