Data De-duplication - Finding The Right Fit For Your Organisation | Articles | FutureGov - Transforming Government | Education

Sunday, 5 February 2012

RESOURCE CENTRE

Tax and Revenue Management: A government’s lifeblood

IT has provided the opportunities for governments to remodel the entire process of tax collection over the last decade. It is, however, a continuously evolving process and governments the world over need to constantly upgrade their tax systems to optimise their revenue workflows.

Unlocking Public Value

A recent SAP study confirmed that those organisations which adopt best practices in the areas of scope and adoption, process standardisation, technology and customer governance, do perform better, and do so as their best practice maturity increases.

Governments and Socialising

The advent of social media has seen governments hopping onto the bandwagon in a bid to further engage citizens.

Government Data Management, Technology

Data De-duplication - Finding The Right Fit For Your Organisation

9 March 2010

With global drivers such as rising IT operating costs, exponential data growth, regulatory compliance, strict service level agreements, and shrinking backup windows, enterprises around the world are forced to re-evaluate their data protection methodologies. In addition to the aggressive virtualisation strategies being adopted by many global companies, IT managers are faced with an entirely new set of challenges – on top of their current concerns about data protection residing at remote or branch offices.

Photos

View photos

Related Categories

From this Section

SPOTLIGHT

A present day key challenge that has begun to impact backup performance is the amount of data that must be protected within the available backup window. Traditional solutions are inefficient in most cases because they back up everything including duplicate data files and sub-file data segments that exist across servers, desktops, laptops, and offices worldwide, day upon day.

More importantly, the impact is extremely severe in a virtualized environment and remote offices where each virtual machine represents an individual backup job, often with concurrent or overlapping backup windows, and includes redundant operating system, application and file data. As a result, backups for virtual machines can often overrun backup windows and tax shared resources, leaving data unprotected and creating management issues for backup administrators. Any cost savings gained from hardware consolidation is now offset by the exposure of being unable to protect one’s environment under current Service Level Agreements.

Also, by backing up duplicated data, traditional solutions also increase cost due to the extra storage bandwidth, capacity and management required. Thus, IT managers are looking towards data de-duplication as a simplified approach and solution to managing data growth.

Using Data De-Duplication for Backup

An effective data de-duplication product will help organisations cope with backing up information stores by removing data that is redundant to economize the storage and disaster recovery requirements for data.

As IT managers are already aware, there is tremendous data redundancy in backup environments. Effective deployment of data de-duplication allows organisations to protect and restore information at a fraction of the footprint and operation expense of their current backup storage infrastructure.

Some of the business benefits derived from an investment in data de-duplication include:

Lower infrastructure costs By eliminating redundant data from the backup, far less infrastructure is required to hold the backup images. Data de-duplication directly results in reduced storage capacities to hold backup images. Smaller capacity requirements means lower acquisition costs, reduced power and cooling costs and the ability to minimize network bandwidth requirements and costs.
Improved data protection

Data de-duplication enables many organisations to create daily full backup images while only consuming a fraction of the storage capacity required for even an incremental/differential backup. Many of these organisations had been forced to do weekly full and daily incremental due to backup window constraints. As a result, restoration and recovery took much longer as each day only represented a partial recovery. De-duplication reduces storage capacity requirements, which permits more aggressive backup policies with improved restore times by giving end-users the ability to have a file system view of a full restore every day.

By reducing the total quantity of backup image size, companies are better able to afford the replacement of traditional tape storage with disk for backup. Backing up to disk enables high-speed, highly reliable backup images – which supports the need for both shorter backup windows and faster recovery times.

However, there are many forms of data de-duplication, and this makes it difficult for organisations to implement only one form of data de-duplication across its business locations. Yet, while definitions may defer, there are three basic forms of data de-duplication that the IT manager will need to be aware of. First is compression, which is a method of reducing the file size. Second is single-instance storage, which is the removal of multiple copies of any file. And third, is sub-file de-duplication, which detects redundant data within and eliminated even when the duplicated data exists within separate files.

Where does Data De-Duplication Occur?
There are some technology considerations when determining the optimal de-duplication solution for the organization. These considerations include understanding if the de-duplication occurs at the information source or at the backup target. Also, it is important to know and consider the appropriateness of in-band or out-of-band architectures for the organisation’s backup environment.

Source-based De-duplication Source-based data de-duplication provides for the elimination of redundant data at the source. This means that data de-duplication is performed at the start of the backup process – before information is transmitted to the backup environment. Source-based de-duplication can radically reduce the amount of backup data sent over networks during backup processes. This is important if there are bottlenecks in the back up process related to networks, shared resources or backup windows. Furthermore, there is also a substantial reduction in the capacity requirements needed to store the backup images.
Target-based de-duplication Target based de-duplication is an alternative to source-based de-duplication. Target based de-duplication happens at the backup storage device. This form of de-duplication does not require users to change their incumbent backup software. Target based de-duplication does require that all backup images are copied to the backup appliance, so target-based backup is not a solution that reduces backup-client-to-target bandwidth requirements.

From EMC’s perspective, the best way to manage and overcome this challenge is to reduce the size of the backup data at the source itself before it being transferred across the network and stored to disk.

De-Duplication Effectiveness

The first decision to make is to decide whether de-duplication is warranted in a particular environment. Many application environments are well suited to data de-duplication and optimally suited to a particular implementation of de-duplication.

However, some environments will not maximize the benefits of de-duplication, and various de-duplication solutions have different de-duplication ratios, which is the size of the data backed up compared to the size of the backup images on disk.

Currently, there are several factors that contribute to the final de-duplication ratio that will affect an organisation. Some of these factors are intrinsic to the de-duplication technology and others are environmental factors. Briefly, some of the common factors that affect de-duplication effectiveness are:
1. Retention period - The longer the data retention period in backup, the greater the frequency of duplicate data already in the backup image store.
2. Ratio of full backups to incremental backups - The more frequently full backups are conducted, the greater the advantage of de-duplication reducing the size of the backup store.
3. Change rate - The fewer changes to the content between backups, the greater the efficiency of de-duplication.
4. Data type - The data from natural sources (audio, images, scans) is highly unique compared to application-generated data (documents, e-mail, presentations). The more unique the data, the less intrinsic duplication exists. File-level de-duplication can still occur, but sub-file level is less likely.
5. De-duplication method - Variable-segment length, sub-file de-duplication will discover the highest amount of de-duplication across an organisation.

Thus, when selecting a de-duplication solution, it is important to clearly understand the environment and backup challenges it represents. Given the situation, a recommended three step EMC de-duplication approach that you may want to consider is:

Step 1: De-duplicate at the highest level of abstraction.

Step 2: De-duplicated as close to the source as practical.

Step 3: Find the most interoperable approach across multiple use cases.

Though our checklists and our recommended three step approach, enterprises will be able to save on bandwidth costs and as a result, creating an affordable and secure way to implement greater protection and recovery, even for offsite office locations. After all, at the end of the day, customers in all market segments are looking for ways to improve service level agreements while reducing the complexity, costs, and risks of managing a backup infrastructure.