# Intent to help some one who are looking for a Process document for Best practises and Process on Windows and/or ESX host patch update process

Purpose

This document provides the process, procedure, challenges, and common risk to

1. Maintain the integrity of Server and its data by applying Operating system and application critical and security updates
2. Establish a baseline process to ensure the Server environment is up to date and compliance with the various standards

This document provides the importance of keeping the system up to date with latest updates and ………

Scope

This process addresses all server environment including Physical platform and its updates (Firmware and drivers), Security updates on all guest Operating system including Windows, Linux, HP-UIX etc, host level updates on VMware ESXi and application level updates on other VMware product on Organization.

Responsibility

Below Team responsible for maintaining a stable up to date server environment with latest updates

• Server Operation Team
• Enterprise Security Team
• Application administration

Team assignments

Server Operation Team

• Patch installation for server systems, including operating system application patches.
• Work with customers when problems arise from patching test systems, or application vendors do not support the latest patches for their applications.
• Direct customers to server exception form when patches cannot be applied.
• Maintain a working knowledge of patches that are deployed in the respective areas

Enterprise Security

• Monthly review of server patch compliance.
• Monthly review of patch exceptions.
• Bi-Weekly review of vulnerability scanning results.
• Apply appropriate security measures to systems that cannot be patched.
• Work directly with systems owners with systems on the exception list.

Application Administrators

• Patching/updating applications on servers. Full responsibility for application patches.
• Working with vendors to obtain application patches when vulnerabilities are discovered.
• Responsible for all patching that is not maintained by the Tech Support Team.

Security Update Process

Baseline and harden

A baseline is a set of configurations for a product or system that has been established as the company standard for building and deploying systems. An application or software baseline should contain the information required to rebuild a system to a desired state. And, more importantly, the baseline should be used to rebuild or deploy a new system to the most current secure state. The baseline should contain all of the most current vendor-released patches.

• Gather and consolidate inventory data on every component of the server environment which is vulnerable, with version and current update level.
• Create a checklist data with each component updated the version and patch level.
• Publish this baseline document across Operation/Security/Application administrators
• Any new provisioning like Hardware/Software/ Application should meet approved bassline
• Regular monitoring of Baseline and update them accordingly
• There should be an approved software collection from a common repository (restrict them install copy setups from other locations) – Published through SCCM
• Any additional software installed on a server should go through Security and version checking (Vulnerability Assessment Tool Scan)

Note: Maintaining the baseline is responsibility of Patch management team (which include Operation/Security and application team). Baseline document should be updated every month post update cycle.

For Example

Date Updated	Component	Items	Current approved Version	Approved and Signed
1/1/2016	Hardware HP Blades	Firmware and Driver	HP SPP Ver x.x	….
1/1/2016	Standard application	NetBackup	7.x	……
1/1/2016	Application S/w	MS office	2010 sp2 or higher	……
1/1/2016	Application S/w	VC++	2010 SP2 or higher	………….

NB: The process of updating the baseline can be handled through the standard patch management processes

RISK and RATING of Servers

H R Block servers should be segregate it’s all production system in to 2 categories, based on the risk and criticality rating

This will help in develop a solid and stable test environment for testing the deployment of updates (whether it is hardware or software driver or firmware update)

Mission critical

An environment in which even one hour of downtime will have a significant impact on the business service, and availability is required at almost any price. Examples would be client facing applications or sites where downtime can translate into significant lost revenue and consumer confidence.

Business critical

An environment in which business services require continuous availability, but breaks in service for short periods of time are not catastrophic. Examples would be E-mail servers, File servers, non-client facing application servers.

Develop a Test Environment

Every test environment should be a mirror of production environment, at least on all mission critical applications defined as mentioned above.

Application owners and team should provide with sufficient time to test the application functionality and operations for a week time at least before applying updates across mission critical application.

It is good practice to maintain a pre-staging update schedule before mission critical server update, time application team to work and come back with any possible issue with the update or Security team to address known vulnerabilities from Non-Production system.

The Patch Management Process

The patch management process has four major phases, which is mandatory to build a strong updates process.

1. Assess
2. Categorize
3. Evaluate and Plan
4. Deploy

Assessment

Assessment phase include gathering information about the environment and Assessing the required security updated based on

1. Security Threats
2. Vulnerabilities identified
3. Criticality

Assessment phase has two components, inventory, and Baselining

inventory

Create and maintain a hardware and software inventory for Organization server environment-  We should create and maintain a clear inventory record of all hardware equipment and software packages, along with version numbers of those software packages most used within the organization. This inventory will help better monitor and identify vulnerabilities and patches that are applicable across the Server environment.  (Software inventory can be part of Baseline Checklist)

Baselining

Mentioned previously in this document as a prerequisite to patch management, baselining requires that you perform updates to maintain standard deployments for the servers and applications in Organization server environment.

Standard configurations should be created and maintained for every component of Server. Standardized configurations can simplify the patch testing and application updating process, and will reduce the amount of time/resources for patch management.

The tasks for providing ongoing assessment are:

• Inventory existing computing assets
• Assess security threats and vulnerabilities
• Determine the best source for information about new software updates
• Assess the existing software distribution infrastructure
• Assess operational effectiveness

Identification/categorization Phase

During the identification phase is to discover new software updates in a reliable way, determine whether those updates are relevant to our production environment, and determine whether an update requires a normal-process or emergency deployment.

All Microsoft/third-party or hardware vendor has their own security bulletin to alert the end users about the major security threat, security, and feature updates subscribers via email.

Once we understand the details about the patch and we have identified that the patch applies to our server, download the actual source files. SCCM will download MS patch source files for you either based on a set schedule or through an updating mechanism. Any third-party application updates should be manually downloaded and create custom package for SCCM for make it available on repository to publish across applicable servers.

Physical server HP (blade and Rack server) drivers & Firmware and Cisco Blade Driver update on windows servers can be updated through custom created packages through SCCM.

Cisco Firmware (Windows & Linux servers) pushed and make it available on UCS central and acknowledge based on the Production and Non-Production maintenance schedules

ESX host updates running and make it available on Update manager in every alternate week, tested and deployed based on normal patching procedure. No direct impact to the Mission critical or business critical application due HA and DRS facility available on VMware Non-Production and production Clusters

Prior to full-blown testing, verify that the files are good and that they install and uninstall correctly, as prescribed in the security bulletin. Review all the options available for deploying the patch, and document these options before entering the evaluation and planning phase.

Evaluation and Planning

The goals during the evaluation and planning phase include:

• Make a go/no-go decision to deploy the software update
• Determine what is needed to deploy the update
• Test the software update in a Non-Production environment to confirm that the update does not impact business-critical systems and applications

Patch testing is vital to determine whether a new patch will affect the normal operation of any existing software.

All Non-Production servers will be scheduled to get patched immediate Saturday after Microsoft windows update security bulletin release Tuesday (Second Tuesday of every month)

ESX hosts, non-Production clusters scheduled to start at First Monday of every alternate month.

In addition to identifying any unintended problems, patches should be tested to ensure that they have fully patched the vulnerability in question or corrected the reported issue if any. This can be accomplished by:

• Checking that the files or configuration settings that the patch is intended to correct have been changed as outlined in the vendor’s update bulletin documentation.

• Scanning the non-production server and host with Vulnerbility Assessment Tool scanner that can detect known vulnerabilities. Address them after getting right fix for the same.

Patch Deployment and Verification

Deployment phase has mainly three part

• Deployment preparation
• Deployment of the patch to targeted computers
• Post-implementation review

Deployment preparation

Deployment preparation include notification of deployment plan through the all possible communication channel defined.

Communication is key to a successful deployment of updates across the Server environment

It should be defined to

1. Gather Pre-Production/ Non-production server updates results from application team, which will help in fixing any know issues with updates/hotfix across the server environment before mass deployment across the Production servers (Key for Go/No-Go decision)
2. Gather the result of Security Scanning from Security personnel, a Vulnerbility Assessment Tool report will identify all security vulnerabilities fixed or not fixed. Any open vulnerabilities should be identified and fixed through interim updates schedule on no-production update cycle
3. Gather Management approval on update deployment
4. Notify management/Stakeholders/end-user about the Update schedule and update details
5. 

Deployment of the patch to targeted computers

My Organization uses Microsoft WSUS and SCCM as standard centralized management and deployment methods for Windows Update monthly patching.

VMware ESXi host update will be carried out using baseline created though VMware Update manager, managed through VMware vSphere client

In addition to initiating the deployment, must take care of

• Monitor and report on the progress of deployment
• Handle failed deployments

Windows Update Schedules

• Following the Microsoft “Patch Tuesday,” All non-production Windows servers are first patched on the second Saturday of every month.
• Current month patches will be tested
• All Windows production servers are patched on the third Saturday of every month ()
• Microsoft WSUS and SCCM are the standard centralized management and deployment methods used for monthly patching.

Procedure document  for Windows and ESX host update provide guideline and steps involved.

Post-Implementation Review

The last step in the deploy phase is to gather your deployment statistics, discuss them with your patch deployment team, and document them. Use these statistics to determine:

• Whether the deployment was successful
• Whether you need to tweak any of your processes to ensure better success in the future
• The performance of those individuals with specific tasks in the process

Potential exceptions and problems

 

Appropriate corrective measures

 

Other Documents or Process required

1. Baseline document (Hardening document)
2. Server update Exception form (Server list)
3. Mission Critical and Business server list
4. Hardware and Software inventory (Software inventory can be part of Baseline Checklist)
5. Windows and ESX host update procedure document