The Evolution of the Data Warehouse
View the full video here: WhereScape
Mike Matchett: Hi, I'm Mike Matchett Small World Big Data, and we are talking today about sort of the evolution of the data warehouse. You know, everything is getting faster. Everything is moving more along the lines of being describing code and being dynamic and being something you can you can push out there in the environment and change that way. Data warehousing is no exception. Apparently, where escape has been on top of this for for a number years. And they are here today to explain a little bit about their solution for automating data warehouses. I've got Neil Barton, who's a CTO. Welcome, Neil.
Neil Barton: Oh, thank you for having me. Glad to be here. Okay.
So the idea of a data warehouse to me sounds usually like it's got a lot of terabytes of data. Data has gravity. It's sitting there. It's hard to build. It's hard to change. There's a lot of people involved. And it's you know, it's an anchor it's a good anchor for a lot of companies and anchors, a lot of business processes. But it's a big anchor. You guys are seeing and providing a solution that allows somebody approached that idea of data, whereas differently, right?
A Different Approach to Data Gravity
Neil Barton: Yeah, absolutely. I think you hit the nail on the head. I mean, the conventional way of building a data warehouse from your design discovery to a logical and physical model and then out to each CEO code and the ANC and deployment operations the conventional way and the waterfall WAIS always been very slow and risky. And what we've done with our product is automated, the full lifecycle from me to win. So off of a meta data framework allowing you the ITP people to actually do the discovery and design, working with the business hand in hand and then automating the the rote and repetitive aspects of building and delivering a data. Well so from the business side of the house I get the time to value and improve significantly. I can go from conception to delivery much faster. From the I.T. side, it's about reducing the time, the cost and the risk in delivering these these projects to the business and allowing your I.T. resources to your valuable I.T. resources to work with the business, to deliver value for them and have the tool do the heavy lifting in the road, competitive work.
Mike Matchett: So. So the data warehouse. Would you guys allow someone to describe a data warehouse more than actually build a data warehouse and then your tooling takes that metadata description and builds the data warehouse or or keeps it current? Right.
Neil Barton: Yeah, I think it's a good idea. We start from the allow you to describe it in a logical model, then you convert that into a physical model and in the tool generate all of the tables and structures, all of the indexes and tables for the database platform and the E ulti code for doing all the data pipeline processing. We'll push that down on to the platform that you've got in place. So if I've got charity, it will generate new tier data. Native code benn's very familiar to again, if I go, you know, Oracle SQL Server will be native for those platforms. And in today's world, with the cloud being so prevalent to play up into a redshift or a snowflake, we generate code for those platforms. Again, native codes. So it's not black box I.T. resources to be very comfortable with what they see and what they what they can use within their environment.
Mike Matchett: You can use code reviewers. You can actually insert changes or instrumentation if you needed to.
Neil Barton: Absolutely. And we've got I mean, all of our codes is pretty optimized. The other Boks, but you can inject yourself at any point and make tweaks to the code and add any additional customizations that you may want. It's not hidden away or locked down. It's very open and accessible to your your developers.
Mike Matchett: So it sounds like this makes a data warehouse project. I mean, I've talked to people where it can take easily a year, six months at a minimum.
Mike Matchett: I think as we were talking, there's a high failure rate in a lot of data warehouse projects. People don't know to. What do you guys bring that down to?
What About High Failure Rates in Data Warehouse Projects?
Neil Barton: When I wanted to bring a, you know, like Sydney and projects arranged from six months to a year or longer for the convention where we can bring delivery down into days and weeks by automating all that code. And into what it frees you up to do is actually work with the business and iterate on the design. So a lot of the reasons for failure were they took a requirement from the business and they come back six months later and nine months later and say, hey, we've built what you asked for. The business said, that's great, but that's not actually what we want or the businesses actually change. So what we allow you to do is literally sit down next to the business person and actually iterate on the design and understand what may be different in your source system. Make changes, iterate, make changes and then deployed it in production. Then you can do that in the days and weeks rather than months and years.
Mike Matchett: And you didn't use the word. I'll throw it out there. It's kind of a buzzword, but agile comes to mind making the data warehouse agile. Right. You do iterations and and approach this from a design driven perspective rather than a an infrastructure. What can I build perspective? Right.
Neil Barton: Absolutely. And I think a lot of times you work with the business and then once they start to see the data, they understand maybe there's some some weirdness within the day that they need to account for or they didn't realize there's other stuff out there. So they can start to actually incorporate that in the design rather than waiting six months to find out. It's it's not what they wanted.
Mike Matchett: So it sounds like this is a very good way for for a standard requirement for data warehouse to become something I.T. can bring to the business and really start to add value rather than. Can I deliver? Can I meet the great minimum expectations? I can start to say, how can I really optimize this and accelerate things? What? What other kind of values does this help I.T. with? There's lots of things out there that I'm concerned with.
The Data Warehouse Lifecycle
Neil Barton: So, yeah, I think we've talked about the design and the development of a data warehouse, but we escaped really extends it to the full lifecycle so we can do the deployment and operations aspect and deploy that code through coupé and test in a new production. And also the operations maintain jobs and schedules to do the loading and processing all their good governance around logging, auditing, what actually happens from my processing. And then yeah, any good successful data warehouse environment is going to need to evolve once it's a living, breathing organism. So I'm going to want to extend in any subject areas in the fact that we have a meta data framework that describes everything that's going on allows us to do things like provide impact analysis and track back and track for diagrams. So that I understand when I enhance my way house six months from now, what's actually going to be impacted again comes back to reducing the risk on making ongoing changes, which the conventional data warehousing and the second order cost the way you really get get hit from a maintenance standpoint.
Mike Matchett: Yes, I was talking to somebody at a recent conference who a big insurance company, and they are stuck, I think literally nine revisions back on their primary software because they can't even make the minimal amount of change necessary to even bring to currency, much less add some new things to their environment. So, I mean, that sounds like a really good thing. But also reminds me of people now facing compliance requirements and going like, oh, I built this thing five years ago. I built one last year and now I've got new requirements. What I do this year, it sounds like this is the kind of approach that you would really need to to handle compliance properly.
Neil Barton: Absolutely. And what am I doing in GDP as a good example? What am I doing with the data? What PDI data do I have? What what transformations are making on the data capture net in, say, don't lose the tribal knowledge when your developers leave and then providing them logging, auditing in lineage around what's actually going on. It's critical for the good governance of these environments historically not been done because if I'm building you by hand documentation that's always facing kicked off the sled to make the project deliverables doubling it by being able to automate it with tools like Westgate, that debt documentation is really free. Just click of a button.
Mike Matchett: Okay. So I'm also seeing that you guys aren't just also stuck back in the traditional data warehouse, but you over the years have brought a bunch of new technologies to the table here and into the same realm. And I'm just looking down here, you've got data lakes, you've got change in capture, streaming data, data vault. What are people consuming this? Are you able to help people now get in in front of some of this runaway train of semi structured, unstructured data, streaming data and and keep up with it? Yeah.
Neil Barton: We've found a huge demand the last couple of years with since since the data and the business seeing value. And that's ahead away from an I.T. standpoint, delivered it to the business side. Streaming addition is a huge value add there. And then what we see now with this unstructured data, the semi structured processing, the data scientists, a lot of companies are wanting to modernize their data warehouse infrastructure. Whether that is bringing new platforms or what we see a lot of go into the cloud. Right. I reduce the the I.T. or the operational costs and management overhead, but I've still got to build my data warehouse. So part of the modernization strategy we're seeing is a move in the cloud is actually go, what a nation first as well. So I'm gonna modify, modernize it my infrastructure by going cloud, reduce the the operational, the I.T. This is Edmundo, the head from my database platforms. But I want to automate that data infrastructure as well. So the automation is a key aspect for them to be able to deliver value for that business over the long haul going forward.
Mike Matchett: I mean, i2 automation is, you know, the top Warner's top two I.T. drivers we see continually year after year whenever we do surveys or ask people about what are the initiatives, it's I.T. automation. Everybody knows that they need a data data analytical capability. Most people have a data warehouse of some kind already. So it sounds like your you're offering quite an interesting way to make that agile and make it. You know, we just really at the time to value of a data. Absolutely.
Neil Barton: And that then is that that is the key thing from the business standpoint. We're used to improving the time to value. And from the I.T. standpoint, coupled with that is reducing the time, the cost and the risk for them to deliver those projects be successful.
Mike Matchett: Awesome. Awesome. I love that your tools are available on target. No different platforms so people can rethinking that way, too. Where should someone go to find more about this, I'm assuming on your Web site?
Neil Barton: We decided it would be the best place to go. We've got a number of articles in podcast copies up there. And so I would go to the website diskette dot com and we've got a lot of information.
Mike Matchett: Awesome. Awesome. Well, thank you for being here. Today, Neal, and explaining this to us, I'm sure a lot of people are going to want to see a little bit more about this in person when they start to hear like, I can automate this thing, that'll be great. This thing is a cornerstone of our business. So thank you for being here today.
Neil Barton: Appreciate your time. I was hoping to talk about what we can do and how we can add value for businesses.
Mike Matchett: All right. And thank you, guys. Stay tune. I'm sure we're going to cover more about I.T. automation in all sorts of aspects and other upcoming segments. Take her.