I have worked on data warehouse projects in many different environments for over a decade. None have been perfect. Issues I have seen range from cubes that take 12 hours to refresh at the end of every month, to business requirements changing so much that the data warehouse was scrapped before it was complete (and sometimes, with a new data warehouse project starting immediately afterwards with the new requirements!)
There are many reasons why data warehouse projects fail, here I look at a few of them.
It Takes Too Long to Deliver
Designing and building a data warehouse is not something that will be completed in a few weeks. It is a huge investment for most businesses, and it is unreasonable to not provide some of the benefits sooner rather than later. If it takes 2 years before the business users see any value from the data warehouse project they will have lost interest, or worse, they will have come up with their own ad-hoc reporting solutions in order to answer the questions they have about how the business is doing.
A phased approach is the sensible option. The project should be broken into smaller chunks. This allows something to be delivered to the business sooner, rather than waiting for the complete data warehouse. This obviously requires some careful planning, and it often seems counter productive. It might take three years to build a bike, then a moped, then a car – rather than two by building the car straightaway – but the impact is huge. Stakeholders are more engaged because they can easily move from A-B quicker on their bike or moped than on foot, even if it’s not their dream car – yet.
Lack of Support from the Business
Building a data warehouse is s slow and expensive process. Just because the decision-makers approved at the start of the project does not mean that they will continue to support the project. If they don’t start seeing results they will rapidly lose interest in the new shiny data warehouse that is costing them a fortune. It is important to keep people from every level of the organisation involved in the project.
It is tempting to sit in a corner coding all day, but that is the worst thing you can do. Keep other areas of the business informed of your progress. Everybody in the business should know the high-level goals of the project.
I have worked on teams that refused to let other areas of the business look at the data in the data warehouse – what were they thinking?! The data warehouse belongs to everyone!
Not Involving the Correct People
Designing and building a data warehouse is not just a technical project. Well before any code is written people from all sorts of areas should be involved. It is a business project and understanding the business need is the first priority. The business should be screaming out for a data warehouse, not a team in the IT department suggesting to the rest of the business that there is a need for one. A successful data warehouse starts with understanding what questions people will be asking of the data. Not just today, but in the future too – the data warehouse should be designed and built alongside the business strategy to ensure that it will be relevant in the future.
Data Quality Issues
Much of my experience of data warehouse projects has been in the financial services sector where the business users demand that the reports coming out of the data warehouse are 100% accurate. If a data warehouse is delivered to the business that does not reconcile with other, trusted reports you might as well hit the delete button immediately. It is crucial that reconciliation and testing takes place at every stage of the data load. And, get your hands on those trusted report that already exist, and use them in your reconciliation process, or prove them wrong!
Lack of User Engagement
The data warehouse is only as good as the decisions being made by the business users relying on it. If nobody is using the data warehouse it is a failure. Most business users will be wary when the data warehouse they have heard so much about finally arrives. They will require training – this is a new way of talking to the data for them. Training needs to use business language, not technical jargon. When users do become engaged make it easy for them to throw more questions at the data warehouse. Adding requests for new reports to an ever-growing backlog is not going to satisfy the curiosity you have just encouraged. Make sure you offer a self-service solution too, where users can create their own reports.
As a developer there is nothing more satisfying than seeing the product of months of work being used as intended, especially if that product is helping the business be more profitable. Keep an eye on how the data is being used, ask people what decision they are making. Show as much interest in the product after it has been delivered as you did when it was being built – you will learn loads for your next data warehouse project!