Agile DW/BI: challenges and solutions

Convener: Lynn Winterboer Participants: Don MacIntyre Andrew Webster Grant Meyer Dave Limbaugh Jake Calabrese Monica Ortenzio Tom Austin Sateesh Nagilla Leslie Morse Typical Issues DW/BI teams have with agile adoption: – Design: Trained from youth to design for answering future unknown questions – Testing: Typically don’t have strong testing discipline to begin with, let alone rarely do automated testing, and testing tools aren’t often geared toward databases. – Test Data: Regulatory limitations on copying data from production to test environments (need to find ways to obfuscate the data to do this). – Documentation: Always struggling to keep the data dictionary up to date. “that was some knitting the kittens got into”. – “We’re different” – Not used to leveraging mainstream software techniques like continuous integration and automated testing (“DW folks are to the rest of IT what the rest of IT is to the Marketing team”) – Analysis: …Seeking to find “one source of the truth” – analysis takes forever. …Analysis in data projects is like building a skyscraper in London – never know when you’ll hit the remains of a Roman building, all construction comes to a stop while you figure out what it is and if it needs to be historically preserved. Solutions discussed: – Getting people past “no you can’t” and experiment, then they CAN – Honor the data team’s concerns – they have some big challenges that are legitimately hard to overcome… – Dig at “why” they struggle with the mindset of evolutionary architecture. Similar to issues BAs have – their fundamental value proposition and training goes against it. – Automated testing: …DbFIT …Custom testing mechanism: SQL that runs and builds tables, populates tables, runs ETLs, leverages excel spreadsheets – Platforms out there that allow you to do data obfuscation whereby a person gets a different name and remains that different name in the DB going forward. – Parallel production system that only a subset of people can access. – Blue/green switching, A/B switching (multiple production platforms) – A special platform that only a few key beta customers can access. – Documentation is essential – what have you “filled out” and what haven’t you! – Use a wiki for documentation – Must, Should and Could documentation. – Self-documenting code – Informatica MetaData Manager – works with Erwin. – Specification by example: Specification that’s testable without further interpretation Lynn shared the attached handout containing 2 of her favorite slides from Ken Collier’s presentations. Thank you, Lynn Winterboer Agile Analytics Coach 303-859-6233 > pdf iconAgile Analytics Handout.pdf

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>