# Intent to help some one who are looking for a Process document for Best practises and Process on Windows and/or ESX host patch update process
Purpose
This document provides the process, procedure, challenges, and common risk to
- Maintain the integrity of Server and its data by applying Operating system and application critical and security updates
- Establish a baseline process to ensure the Server environment is up to date and compliance with the various standards
This document provides the importance of keeping the system up to date with latest updates and ………
Scope
This process addresses all server environment including Physical platform and its updates (Firmware and drivers), Security updates on all guest Operating system including Windows, Linux, HP-UIX etc, host level updates on VMware ESXi and application level updates on other VMware product on Organization.
Responsibility
Below Team responsible for maintaining a stable up to date server environment with latest updates
- Server Operation Team
- Enterprise Security Team
- Application administration
Team assignments
Server Operation Team
- Patch installation for server systems, including operating system application patches.
- Work with customers when problems arise from patching test systems, or application vendors do not support the latest patches for their applications.
- Direct customers to server exception form when patches cannot be applied.
- Maintain a working knowledge of patches that are deployed in the respective areas
Enterprise Security
- Monthly review of server patch compliance.
- Monthly review of patch exceptions.
- Bi-Weekly review of vulnerability scanning results.
- Apply appropriate security measures to systems that cannot be patched.
- Work directly with systems owners with systems on the exception list.
Application Administrators
- Patching/updating applications on servers. Full responsibility for application patches.
- Working with vendors to obtain application patches when vulnerabilities are discovered.
- Responsible for all patching that is not maintained by the Tech Support Team.
Security Update Process
Baseline and harden
A baseline is a set of configurations for a product or system that has been established as the company standard for building and deploying systems. An application or software baseline should contain the information required to rebuild a system to a desired state. And, more importantly, the baseline should be used to rebuild or deploy a new system to the most current secure state. The baseline should contain all of the most current vendor-released patches.
- Gather and consolidate inventory data on every component of the server environment which is vulnerable, with version and current update level.
- Create a checklist data with each component updated the version and patch level.
- Publish this baseline document across Operation/Security/Application administrators
- Any new provisioning like Hardware/Software/ Application should meet approved bassline
- Regular monitoring of Baseline and update them accordingly
- There should be an approved software collection from a common repository (restrict them install copy setups from other locations) – Published through SCCM
- Any additional software installed on a server should go through Security and version checking (Vulnerability Assessment Tool Scan)
Note: Maintaining the baseline is responsibility of Patch management team (which include Operation/Security and application team). Baseline document should be updated every month post update cycle. |
For Example
Date Updated | Component | Items | Current approved Version | Approved and Signed |
1/1/2016 | Hardware HP Blades | Firmware and Driver | HP SPP Ver x.x | …. |
1/1/2016 | Standard application | NetBackup | 7.x | …… |
1/1/2016 | Application S/w | MS office | 2010 sp2 or higher | …… |
1/1/2016 | Application S/w | VC++ | 2010 SP2 or higher | …………. |
NB: The process of updating the baseline can be handled through the standard patch management processes
RISK and RATING of Servers
H R Block servers should be segregate it’s all production system in to 2 categories, based on the risk and criticality rating
This will help in develop a solid and stable test environment for testing the deployment of updates (whether it is hardware or software driver or firmware update)
Mission critical
An environment in which even one hour of downtime will have a significant impact on the business service, and availability is required at almost any price. Examples would be client facing applications or sites where downtime can translate into significant lost revenue and consumer confidence.
Business critical
An environment in which business services require continuous availability, but breaks in service for short periods of time are not catastrophic. Examples would be E-mail servers, File servers, non-client facing application servers.
Develop a Test Environment
Every test environment should be a mirror of production environment, at least on all mission critical applications defined as mentioned above.
Application owners and team should provide with sufficient time to test the application functionality and operations for a week time at least before applying updates across mission critical application.
It is good practice to maintain a pre-staging update schedule before mission critical server update, time application team to work and come back with any possible issue with the update or Security team to address known vulnerabilities from Non-Production system.
The Patch Management Process
The patch management process has four major phases, which is mandatory to build a strong updates process.
- Assess
- Categorize
- Evaluate and Plan
- Deploy
Assessment
Assessment phase include gathering information about the environment and Assessing the required security updated based on
- Security Threats
- Vulnerabilities identified
- Criticality
Assessment phase has two components, inventory, and Baselining
inventory
Create and maintain a hardware and software inventory for Organization server environment- We should create and maintain a clear inventory record of all hardware equipment and software packages, along with version numbers of those software packages most used within the organization. This inventory will help better monitor and identify vulnerabilities and patches that are applicable across the Server environment. (Software inventory can be part of Baseline Checklist)
Baselining
Mentioned previously in this document as a prerequisite to patch management, baselining requires that you perform updates to maintain standard deployments for the servers and applications in Organization server environment.
Standard configurations should be created and maintained for every component of Server. Standardized configurations can simplify the patch testing and application updating process, and will reduce the amount of time/resources for patch management.
The tasks for providing ongoing assessment are:
- Inventory existing computing assets
- Assess security threats and vulnerabilities
- Determine the best source for information about new software updates
- Assess the existing software distribution infrastructure
- Assess operational effectiveness
Identification/categorization Phase
During the identification phase is to discover new software updates in a reliable way, determine whether those updates are relevant to our production environment, and determine whether an update requires a normal-process or emergency deployment.
All Microsoft/third-party or hardware vendor has their own security bulletin to alert the end users about the major security threat, security, and feature updates subscribers via email.
Once we understand the details about the patch and we have identified that the patch applies to our server, download the actual source files. SCCM will download MS patch source files for you either based on a set schedule or through an updating mechanism. Any third-party application updates should be manually downloaded and create custom package for SCCM for make it available on repository to publish across applicable servers.
Physical server HP (blade and Rack server) drivers & Firmware and Cisco Blade Driver update on windows servers can be updated through custom created packages through SCCM.
Cisco Firmware (Windows & Linux servers) pushed and make it available on UCS central and acknowledge based on the Production and Non-Production maintenance schedules
ESX host updates running and make it available on Update manager in every alternate week, tested and deployed based on normal patching procedure. No direct impact to the Mission critical or business critical application due HA and DRS facility available on VMware Non-Production and production Clusters
Prior to full-blown testing, verify that the files are good and that they install and uninstall correctly, as prescribed in the security bulletin. Review all the options available for deploying the patch, and document these options before entering the evaluation and planning phase.
Evaluation and Planning
The goals during the evaluation and planning phase include:
- Make a go/no-go decision to deploy the software update
- Determine what is needed to deploy the update
- Test the software update in a Non-Production environment to confirm that the update does not impact business-critical systems and applications
Patch testing is vital to determine whether a new patch will affect the normal operation of any existing software.
All Non-Production servers will be scheduled to get patched immediate Saturday after Microsoft windows update security bulletin release Tuesday (Second Tuesday of every month)
ESX hosts, non-Production clusters scheduled to start at First Monday of every alternate month.
In addition to identifying any unintended problems, patches should be tested to ensure that they have fully patched the vulnerability in question or corrected the reported issue if any. This can be accomplished by:
- Checking that the files or configuration settings that the patch is intended to correct have been changed as outlined in the vendor’s update bulletin documentation.
- Scanning the non-production server and host with Vulnerbility Assessment Tool scanner that can detect known vulnerabilities. Address them after getting right fix for the same.
Patch Deployment and Verification
Deployment phase has mainly three part
- Deployment preparation
- Deployment of the patch to targeted computers
- Post-implementation review
Deployment preparation
Deployment preparation include notification of deployment plan through the all possible communication channel defined.
Communication is key to a successful deployment of updates across the Server environment
It should be defined to
- Gather Pre-Production/ Non-production server updates results from application team, which will help in fixing any know issues with updates/hotfix across the server environment before mass deployment across the Production servers (Key for Go/No-Go decision)
- Gather the result of Security Scanning from Security personnel, a Vulnerbility Assessment Tool report will identify all security vulnerabilities fixed or not fixed. Any open vulnerabilities should be identified and fixed through interim updates schedule on no-production update cycle
- Gather Management approval on update deployment
- Notify management/Stakeholders/end-user about the Update schedule and update details
Deployment of the patch to targeted computers
My Organization uses Microsoft WSUS and SCCM as standard centralized management and deployment methods for Windows Update monthly patching.
VMware ESXi host update will be carried out using baseline created though VMware Update manager, managed through VMware vSphere client
In addition to initiating the deployment, must take care of
- Monitor and report on the progress of deployment
- Handle failed deployments
Windows Update Schedules
- Following the Microsoft “Patch Tuesday,” All non-production Windows servers are first patched on the second Saturday of every month.
- Current month patches will be tested
- All Windows production servers are patched on the third Saturday of every month ()
- Microsoft WSUS and SCCM are the standard centralized management and deployment methods used for monthly patching.
Procedure document for Windows and ESX host update provide guideline and steps involved.
Post-Implementation Review
The last step in the deploy phase is to gather your deployment statistics, discuss them with your patch deployment team, and document them. Use these statistics to determine:
- Whether the deployment was successful
- Whether you need to tweak any of your processes to ensure better success in the future
- The performance of those individuals with specific tasks in the process
Potential exceptions and problems
Appropriate corrective measures
Other Documents or Process required
- Baseline document (Hardening document)
- Server update Exception form (Server list)
- Mission Critical and Business server list
- Hardware and Software inventory (Software inventory can be part of Baseline Checklist)
- Windows and ESX host update procedure document