Best Practices in Data Validation

Data Quality is the buzz word in the digital age.

What is data quality and why is it so important?

“Data quality” is the term that is probably hidden but plays an important role in many streams. Data plays a vital role in acquiring a market place, especially in enterprise data management stream.

Data Quality Examples

Following are some examples which emphasize the need for data quality.

• A customer shouldn’t be allowed to enter his age where he has to mention his marital status.
• When a customer enters a store, there is a high possibility that he might miss out his original details to be filled up with the forms, some of it can be in a hurry not mentioning a correct phone number.
• There is also a possibility of the billing staff to wrongly enter the store address as default in place of the customer address which contributes to a bad quality data that gets persisted in the system.

http://www.infotrellis.com/best-practices-data-validation/

Data Warehouse Migration to Amazon Redshift – Part 3

This blog post is the final part of the Data Warehouse Migration to AR series. The second part of the blog post series Data Warehouse Migration to Amazon Redshift – Part 2 details on how to get started with Amazon Redshift, the business and technical benefits of using AR.

1. Migrating to AR

The migrating strategy that you choose depends on various factors such as:

1. The size of the database and its tables
2. Network bandwidth between the source server and AWS
3. Whether the migration and switchover to AWS will be done in one step or a sequence of steps over time
4. The data change rate in the source system
5. Transformations during migration
6. The partner tool that you plan to use for migration and ETL

Learn more: http://www.infotrellis.com/data-warehouse-migration-amazon-redshift-part-3/

MDM Validations – Things to remember when implementing InfoSphere MDM Server

Validation is an important aspect of any application or system. Validations could arise as part of functional requirements (e.g., business rules) or non-functional requirements (e.g., maintain data integrity). Data validation is a process of ensuring that a program operates on clean, correct and useful data.

In any MDM implementation, data validation plays an important role. Since MDM deals with maintaining consolidated view of entities, it is critical that the data stored is valid and meets all business rules.

IBM’s MDM Server comes with a robust and easily customizable validation framework. MDM Server validations can be broadly classified into two types: 1) External validations 2) Internal validations

 First let us talk about what are External validations

External validations are first level of validation. The validation rules are configured in database tables, the definition metadata is retrieved at runtime by validation engines for execution.

http://www.infotrellis.com/mdm-validations-things-to-remember-when-implementing-infosphere-mdm-server/

Master Data Management: Are you flying blind?

How can you govern your master data without knowing your master data?

For many years I’ve been saying that the one thing all MDM clients have in common is that the quality of data in their source systems is not as good as they thought.  Over the past several years I’ve found that all MDM clients have a second thing in common: they are unaware of the quality of data in their MDM hub and they don’t know how the data is changing.  This is surprising since an MDM hub contains your most critical business data that is used in real-time processes and analytics across the organization.  How can you govern your data when you don’t know its trend in quality, how it is being used and how it is changing over time?  This is flying blind.

There are a few contributing factors to this issue.  The first is that MDM products don’t provide capabilities to analyze and report on data.  The second is an MDM hub is not the appropriate place to do this.

http://www.infotrellis.com/master-data-management-are-you-flying-blind/

MDM Validations – Things to remember when implementing InfoSphere MDM Server

Validation is an important aspect of any application or system. Validations could arise as part of functional requirements (e.g., business rules) or non-functional requirements (e.g., maintain data integrity). Data validation is a process of ensuring that a program operates on clean, correct and useful data.

In any MDM implementation, data validation plays an important role. Since MDM deals with maintaining consolidated view of entities, it is critical that the data stored is valid and meets all business rules.

IBM’s MDM Server comes with a robust and easily customizable validation framework. MDM Server validations can be broadly classified into two types: 1) External validations 2) Internal validations

http://www.infotrellis.com/mdm-validations-things-to-remember-when-implementing-infosphere-mdm-server/