Video: Laying the Architecture and Data Foundations for Self-Service Data at Scale | Duration: 3384s | Summary: Laying the Architecture and Data Foundations for Self-Service Data at Scale | Chapters: Introduction to Series (6s), Defining Data Products (394.45s), Data Access Control (766.08s), Combining Access Models (845.42004s), Data Management Fundamentals (940.05s), Metadata Collection Strategies (1069.76s), Efficient Bot Triggering (1941.62s), Dataset Council Strategy (2026.2201s), Encouraging Artifact Reuse (2131.075s), Promoting Data Reuse (2216.5s), Data Quality Strategies (2316.865s), Data Product Management (2478.24s), Monetizing Data Value (2790.095s), AI Agent Foundations (3065.66s), ETL and Agents (3195.95s), Webinar Series Conclusion (3302.81s)
Transcript for "Laying the Architecture and Data Foundations for Self-Service Data at Scale": Okay. Hello, everyone. Welcome. Thank you for so much for joining us for our second session in the series. Whether you were here with us last week or if this is the first session that you've you've attended, we're really happy to have you here. This series is all about how organizations are rethinking centers of excellence, helping empower self-service, keeping government strong, all while figuring out how AI agents fit into the mix. So if you're working on or building or evolving a COE, you're in the right spot. So as folks are still joining in, please feel free to say hi in the chat. Let us know where you're joining in from and what COE challenge you've been thinking about lately. Today, we're shifting gears a bit. The session is focused on building the right architecture and data foundations to support scalable self-service and AI agents. We'll be looking at what it takes on the technical side to support flexibility, government, governance, and speed all at once. And we're so excited to have Jonathan Tudor back with us to walk through it. Jonathan is the director of business architecture here at Dataiku. A couple of housekeeping stuff before we dive in. Please drop your questions into the q and a panel. I know it looks a little gray. It's something that we're trying to fix on, the back end, but just wanted to let you guys know that it works. Chat's open if you wanna share your thoughts or reactions along the way. And, yes, we are recording. You'll get the replay and this oh, the the q and a section, looks normal now. And, yes, we're recording. You'll get the replay and slides in your inbox after this. And so I will pass it over to you, Jonathan. Awesome. Thanks for that. I appreciate the introduction. Let's go ahead and, get started here. Yep. Awesome. So hi, everyone. As we know, I just did a great introduction here, but just to reintroduce, my name is Jonathan Tudor. I'm a director of business architecture here at Dataiku and help our customers really launch, scale, and operationalize self-service programs. If you're joining us here for the first time, welcome to our webinar series on how do we think about COEs, enabling self-service and AI at scale within organizations. Our first webinar was focused on how do you connect the vision and strategy into building a program like this. And today, we'll spend our time focusing primarily on what are some of the data architecture and management foundations that light up scale an ecosystem like this across an organization. So to get us started here today, we'll jump in first to a high level view of, you know, how do you think about data architecture to really enable self-service data or agents at scale? So if you imagine a, user or business, you know, individual in an organization, right, you need to make the process of working with data simple and easy for them. And, actually, much the same applies to working with an agent. Right? You need to make it so it's simple for that agent to execute. So we talk about here are some high level kind of concepts to think about as you think about enterprise data architecture to how you support something like this. So first and foremost is, you know, does your business know where to look for data? In many organizations, there's a desire and need for an enterprise source of truth. Right? How do I find the right dataset for the problem I'm trying to solve? If I have the seven different bill of materials in the organization, which one's the right to use? Right? If I have all these different datasets spread across the company in different ERPs, where do I start? So this idea of enterprise source of truth is really key. Where does data reside? How do people find that data? Next, you know, how do employees easily get access to that data? Right? Is it something that goes through a series of workflows and steps to request access? Do they have to go through maybe an IT team member whose full time job is not doing that work? So what do employees have to go through in order to access that data? Certainly, that'd be true for an agent as well. How do you give agents access to those datasets? Now importantly, where do the new datasets that are created reside? If you have a large self-service community building out datasets, they have to write them somewhere. If they can't write them somewhere, they can't be used by other parts of the organization or reused. Right? And again, some of our ideas apply to agents. Agents have outputs where those outputs go within the organization. And then finally, now does the architecture itself support your data processing at scale? So are you resending all of your processing through a single server and kinda causing a bottleneck? Are you doing more, you know, parallelized compute across maybe, say, a data lake or something like that? There are different ways to structure that, but you need to ensure that your processing is in place so that your user really isn't stopped by something not working, and similarly your agent. So on the now kind of thought process from a Dataiku perspective, kinda look at this bottom left box to talk about this idea of how do you start to connect data across the organization. And there's really kind of two challenges there. One, you wanna make sure your data is not siloed. You don't want different warehouses and data sources across the company that can't be accessed easily from one spot. That'll slow down your user base quite a bit and really lengthen the amount of time it takes to get to an analytical output. Similarly, if you were to take all of your processing and just push it through one Dataiku server instance, that's not gonna scale very well. So you wanna think about ways of scaling that processing. So from a best practice standpoint, we usually look at two things. First is, how do you kinda build that plane for the enterprise source of truth? In modern day architectures, that's done in really two ways. One is the idea of a data lake or data lake house. We are physically instantiating the data in one location and then allowing access to that data and work on top of that data happen within that physical kind of data lake environment. The other option is utilize a data mesh that looks across a broader enterprise data landscape and virtually connects those datasets into one virtual plane for enterprise data that then you work off of and then write data back to. In both those models, though, you'll have an enterprise view of all of your data that's happening, and you'll be able to understand when new datasets are being built or how they're being created. The keyword being observability. Right? You need to be able to see all this activity. Now finally, going to the bottom right, another best practice, how do you manage processing at scale? If you're in a kind of data lake or lake house environment, you can generally configure processing to push down to that environment, something that Dataiku does very well for almost all of its processes. Similarly, you can use containerized compute, where you set up, say, like a Kubernetes cluster, and have these clusters scale up and down based on the compute needs that you have. In either case, now if a user is working in a tool such as Dataiku, trying to process some data, build a new agent, whatever that may be, they won't be hamstrung by not having the resources they need to run that process. So kinda moving away from the the enterprise architecture for a minute. Another kind of key item that comes up is the idea of a data product. Right? What are we actually building as an organization? We're looking to create something. You can think about it similar to another idea of a widget in a manufacturing environment or auto shop floor. We have to define what the object is we're creating so we have the vernacular and the capability to talk about that and then think about how that scales in an organization. Now many companies define data products differently, and what you'll find is if you ask, usually, 10 individuals of the company, how do they define a data product, you'll get 10 different definitions. So best practice from the COE perspective is to come to a common definition across the organization so you can all work together to help build this enterprise type data approach environment. From a definition standpoint, a common definition is that really a data product is a group of, you know, a single or multiple data artifacts that are being used to build a particular outcome. Artifact could be, you know, a data engineering job, an analytical model, could be an AI agent, or it could be a series of those things together that are used to build a product. But the important part is that you've done a series of data centric steps. If you were in the last session, maybe those aligned with the success model to build out that particular data product. So as an organization, some things to think about is now what is the definition of that data product in your business? Is that the same as what's here or is that different? Is that something that the overall business would agree to or do you have different definitions? Say if you're on a supply chain team versus the central IT data analytics team. And then finally, you know, what is that success criteria for when a data product should be built? What is the definition of when it makes sense to spend resources on a data product versus not? Next, you kinda wanna think about what really makes up a data product. Right? Now today's real common things I see are like data visualizations, analytical models, datasets, LLMs, agents, and really anything else that rolls into that process. But you can think about the discrete pieces that build that data product and define those for the organization. So, again, think about what those are for the organization. And then think about, are there only certain components that you want only certain parts of the organization to build, or do you want everyone to build everything? So a common conversation is, you know, do we want to empower the entire business to build their own agents, or do we wanna have a small subsection, maybe a pilot section, maybe a particular group in the IT department? You know, our genuine hope, and as you can tell by the series, is we like to democratize that capability. We think it's the best way to go forward. But it also can make sense sometimes to take smaller steps. So it's really important to codify upfront who do you want doing what so that people have clear guidance around that. Next to think about is, you know, as these data products mature, you know, who supports them? So something that can happen is if you really start to scale, you know, self-service in an organization, you can have users build critical processes, but not have maybe the right operational support around those processes. A good example I've seen in my career, someone built a particular reporting and and analytic process for a shop floor. The individual got sick without for a week, and that same shop floor cannot ship the product for a week because they were out sick. In that model, that's an operational challenge or something you don't wanna have from a consistency standpoint because you're not relying on a single individual. So So what should happen in that case instead is if you look at the criticality of those products being built in self-service and look to understand when do I need different support criteria or different levels of maturity around them. So some ways to think about that are, you know, what are the maturity levels of the data product? You'll see here on the right side of the screen, with this POC pilot product and critical product, a very common format for how this is defined. And you can think about this largely like an innovation funnel. So within a business, we might call something a proof of concept when it's a, you know, new idea that's got five or less users. We're just trying something out, not even really automated. Maybe that data product becomes a pilot once we start to start automating it and we start to see does the business really adopt this. Does this change how the business is doing its work? Right? And that could be anywhere from maybe five to 49 users. The magic number I usually hear is about 50 or so unique users is when something becomes a true product. Right? You now have 50 unique users looking at this data on a regular basis, say on a monthly basis, consuming from it. At that point, you may wanna take that product and move it to more traditional IT support or your data program support. And then finally, the critical product is a similar idea to a product, except you have more of that leadership buy in saying this is critical to the business. Now your definitions again might not be the same, but the idea is to capture that innovation and then look to kinda highlight where that maybe shadow IT would have been in the past. You get it more of a spotlight to say, hey. This is important. Let's bring it into our IT and data management teams. Now a couple of things to think about as well. I already gave definitions of unique users. You may have different definitions to decide when something moves to this level. It could be quality of the model, you know, a certain time period of success with the model. There's many different ways to think about that. And then finally, importantly, what is your RACE? Who supports what? Now does your IT support team only support the operations of the platform that these models run on, or do they actually support the operations of the models on a nightly basis? Again, how do you think about that? And then finally, on the far right here is the idea of how do you really document, these types of data products within the organization. So as an example, if you wanna drive reuse, you have to find get people would actually find what's already been built. So there are a few different ways to do this. A common approach is using Dataiku itself and directly highlighting reusable Dataiku projects on the home page. Think about a marketplace concept. Another common approach is simply using a Wiki. If you've used Confluence or something like that before, similar idea. You can use your enterprise data catalog and show your data products there. Again, the driver use. Or if you have a CMDB system, think of, like, a ServiceNow or maybe ITIL processes. You can also look at defining your data product assets there. The important part again is to have consistency and really standards are how you think about that. So the questions you should ask are, you know, where will users find these existing data products? What types of information or metadata do we wanna capture? Could it be, you know, what a line of business the products for, the criticality, etcetera? And then, you know, you also may wanna think about, can users provide input on the success or maybe not success of the data product? I find this to be pretty powerful because as you see a product become more successful, have maybe more uploads from users, that causes more consistency and interest around the product itself. So kinda switching gears here a little bit, we've now talked about the broader architecture concepts or how you can get observability into what's happening, and we talked about defining what is actually being built in the environment. Another key element of this, though, is how do you actually get access to the underlying data? Right? This is a very big challenge. Almost all organizations need to balance the ability to enable with the ability to kind of protect, right, or keep things safe. So with that in mind, we kinda think about what are some of the data access control considerations in this approach, and this could apply whether you're on a enterprise data lake environment or a data mesh. Same idea largely applies. There's really three different types of access control that you can use First is simply discretionary access control. I asked the data owner, can I have access? Data owner says, yes. I get access to that data. Another example is role based access control. I have a certain kind of role in the organization. This particular role gets access to these assets or projects, so then I automatically get that access. But I don't necessarily have the ability to request a unique dataset. Then finally, there's attribute based access control, which I'd argue is probably the most mature of these. And what this does is say, okay. I've got metadata and attributes about my data. I've got metadata and attributes about my individuals or humans who will be accessing this. How do I match these together and give dynamic access to datasets? That, of course, requires a lot of metadata management and things like that, but allows you to automate and go even faster. So as we look at organizations, and maybe at the highest level, what we'll see is that there's some no. Some of these models, when they run by themselves, tend to kinda leave some of your users out in the cold, so to speak, and keep them from being able to be as effective as they can be. So So as an example, if you're to use role based access control or attribute based access control only, it can actually really reduce your innovation and actually slow your data product development cycle time. And that's because you have an assumption in these models that you know specifically every dataset for the type of user needs to see, which works for a while. Let's say I have a finance user and you wanna give them access to finance data. Let's say you now wanna start cross correlating that finance data with what's happening in your supply chain, what's happening in your sales and marketing environments. Well, now those users who are trying to pull all these steps together now don't really have an easy way other than getting added to more roles and trying to be parts of multiple roles or attributes in the ecosystem. And that can get kind of messy and is usually relatively slow. So that model can kind of break down. Similarly, direct access control can be just two minute requests. Right? If you have thousands of users requesting access to thousands of datasets and they're all individually being approved, that can take a lot of time. So that's also maybe not necessarily ideal. So the goal that you wanna look for is this idea of how do you combine these together. Can you take an RBAC or ABAC model and combine that with a discretionary access control so that you accelerate quick access to known datasets for particular types of users or users in certain roles, but also have the flexibility to request unique access for those users? In that type of scenario, you still can accelerate your development while also keeping your assets relatively safe within the organization. So now that we've kinda talked about all these items around access, products, how do we think about control? Right? The The next idea is now how do we think about data management? Now how do we really think about this idea of how do we make data easy to find? The baseline need of any type of self-service program is really to find the data, the assets you need first. And that's true whether you're building out a visualization all the way up to building out a new agent. You still need to give those agents datasets to consume, but how do you know which ones do you even get to that agent in order for it to make decisions and take actions? That happens by understanding what your enterprise data looks like and really applying data management to the enterprise data. If you're part of our first webinar, this would be very similar to that kind of search step or the first step within the kind of success model. So if you're not familiar with enterprise data catalogs, these are technologies that take a look at the overall metadata or the columns and rows within datasets in the organization and allow you to put metadata around them. So tag if something is potentially critical, have certain sensitivities to it, what the purpose of datasets is, all these types of things. Right? Within the realm of Dataiku, we have a data catalog built directly into the software. You may also be familiar with data catalogs such as Alation or Caliber. Right? These exist across the market and help you really look at your assets holistically. So if we look at some of these methods around how we make data easy to find, as mentioned, Dataiku's data catalog does this. So the enterprise data catalogs. It's also common to see organizations use wikis again. Right? So define certain datasets for a wiki and point users to go use those datasets. Another item now that we see kinda be as a best practice, but maybe not applied as much, is also how you organize your data. Right? So if you're doing an enterprise data lake or if you're doing a data mesh, how do you organize the datasets within that environment so that they're easy to find? Now do you just directly copy source systems and not categorize what they're part of? Like, this is maybe part of this part of the business or this is this part. You know, if you do that kind of broad application, it's really hard to find datasets. But if you get more clear about how I've organized data, it becomes much easier to find it. So think about that as you structure, you know, your enterprise data environments. Now something else that's pretty important is also how do you collect this metadata? Having done this over ten years now, one of the biggest things I get asked is what's, you know, what's one of the Achilles heels of doing these type of programs? And one of those heels is is that in order to run a data catalog or to capture metadata, it's usually a human endeavor to some extent. And it's really hard to necessarily, you know, get humans or employees to spend a lot of their time putting in metadata. So we have to think about ways that we start to capture that. Now first and foremost is automation. If you can automate a process to capture this type of information, that is ideal. Right? So have your data catalog scanned on a regular basis. Maybe have automatically generated lineage by reading in your jobs from various pieces of software. Think about how you automate this as much as possible. The next kind of approach is applying governance. Right? So think of applying governance and processes, making it a requirement to document certain pieces of metadata if you want to automate a project, or making it a requirement in order to go through a certain toll gate. You can apply governance to ensure humans take those actions. Another view that I've seen that's made a number of successful organizations do is actually have dedicated roles. So hire dedicated individuals whose jobs are to go out to the business and champion the idea of collecting metadata from various divisions and really bringing that into the business context. That can be very, very powerful, and the best organizations I've seen at doing this kind of work have dedicated individuals who work from a central COE team and then work across lines of the business to help collect this type of information. And then finally, an item close to my heart is gamification. I believe if you make something more fun, people are more likely to engage in it. You can certainly do that with data cataloging. You can track what people are contributing. You can put that in the scoreboards. You can build various ways to engage. This is giving out stickers and things like that. It's a great way to engage more people, again, to collecting that critical metadata. Now on the right side here are a lot of examples. So I think I got a question that came in here. I'll pause for a second. Is this a place to protect client protected information as well? So we haven't quite gotten into, you know, breaking out things based on data sensitivity, but, yes, I think this is where you start to do that. You know, a lot of my experience is more based on things like expert control. So So think about the idea of a dataset being residing in a certain country and then needing to keep that data restricted to only residents of that country for visibility. Right? So let's say, you're in Italy and we have Italian export control data. Well, in that scenario, you may separate that dataset and only have Italian export control data in that environment, but also having a global kind of set of data for the business. Well, the same idea can kind of apply here when you look at, you know, client protected information. If you have a certain type of dataset that's client know for a particular set of clients, you can physically separate that data right into a data lake. You could also say, hey. I'm gonna use metadata, and I'm gonna tag my datasets and have a lot of governance around me so I can understand the sensitivity. So I can say, hey. This is client protected metadata. And because this data data is protected, I can only use this on certain projects for certain approvals. And importantly, maybe I can only use it with certain AI agents with certain approvals. Right? Because you can tell that agent, I've got this metadata. If it says, no. You can't look at this, then the agent can't look at that. Right? So that's where I think we really start to see some of the value come in from this type of data management as it applies to AI, is that the metadata we provide can guide the AI's decision making about what data access to use or not. So if you need to protect that client data, you can do it through that method, or you can also open that data up again through the metadata as a trigger. Great question. K. So jumping back here to our slide here for a minute. I won't walk through all these on the right, but this is kind of a short list of some of the recommended metadata that we'll talk to customers about. And it goes from everything from what is a simple way to recognize what the dataset is, what is a business recognizable name. Right? It's kinda hard to look at something and see, here's Oracle table x y z. You have no idea what that means. Might be easy if you call it sourcing data. So with that in mind, these are all examples of things you could collect. Right? And, And, again, they start from more descriptive type items, move to things like data sensitivity, tying to to Byron's question here, on the chat. And they also tie the things around what is the quality of the datasets, what is lineage, what is the use. But, ultimately, the goal is to provide enough context around your datasets that you can intelligently ask your organization and your AI agents to automate or do certain processes without that hand holding. The metadata should provide the context to help them drive innovation. That in mind, let's keep this going. So one thing I like to to dive a little bit deeper on is this idea of, you know, data use classification or AI use classification. And the reason this is important is, again, if you have a large enterprise environment, you're gonna wanna classify how you wanna actually use each of these datasets. So if you think about the recommended recommended types of metadata on the previous slide, one recommendation is this classification. Right? And if you think about a world where you have users doing their own data engineering, their own data prep work, building their own analytics, you're actually gonna wanna classify what that engineer or that dataset looks like within the enterprise data environment because you want people to use the right data. And, actually, the same thing applies to an AI model. If you want those model to use a specific kinds of datasets, you need to understand which ones are meant for use by that model and which ones are not. So a simple example I give here on screen is the idea of defining the use of datasets primarily from a data engineering construct. First, imagine if you're an enterprise data lake or a mesh. Right? You've got your base layer of data. So this might be data coming from your multiple source systems, your ERPs across your business, maybe external systems. But it's usually a replication of that data into an environment. Now similarly, as we go into the world of kinda MCP, right, multiple context protocol, as we start to look at that, that context protocol also kinda works with this idea about mapping to how you use a certain data. So if I have a base layer of data tied to, say, finance, well, maybe I tell MCP this is my finance dataset or my finance agents. So, again, there's just idea of routing that happens at this layer. Now as users start to come in and work with data, they may create different types of datasets or different types of assets. Right? So one to think about is transform. If you're deep in data engineering, you might think about this as a temporary table. Right? It's something that's built as part of a job process, but you don't really wanna reuse it because it's meant to temporarily be there, and it's not meant to be the final use of that dataset. So if you could imagine, tagging something as a transform dataset in your data catalog may tell your users or your models, hey, don't go utilize this data. It's not meant to be production grade. Right? The next layer up might be something like consumption dataset or an AI asset. This means that you built that particular data asset, for the organization to be used for a particular use case or outcome. So it's meant for a certain form of consumption. So think about this as being singular use and maybe not meant for multiple uses. Right? And then finally, one layer above that is this idea of published data. If you're familiar with the world of, like, Databricks or other companies, you may also see think terms like gold or silver layer data. You can think about this as a similar construct. It's best use data meant for the organization around, here's the data you should use to build these particular outcomes if you need this kind of data. And I find this is very powerful, especially the larger an organization is. If you're a hundred year old plus company, you've had many acquisitions over the years, you likely have a lot of different versions of truth around your data. And if you kinda go back to this idea of enterprise source of truth for data, we have to define what the right use data is within that. And published datasets can help you do that. And similarly, a published AI, you know, agent or LLM model will do the same thing. This is the best use agent or LLM within the organization for doing this kind of work. So, again, you can see this idea of highlighting items that are more successful or more reusable. Now one thing that's really interesting thing here is that the idea of, again, that innovation funnel of having new ideas happen, new consumption data sets being built over time as new individuals in the business trap these ideas. What is interesting is if you use observability and you're doing all this within a, you know, environment where you can see all the activity, you can start to track the adoption of these data sets. And as you see certain datasets get more adoption or AI assets get more adoption, you can then look to promote those further and make them more usable. So best practice is to actually leverage the knowledge of your organization to build these items out in a self-service way, and then take the best of these and then move them back into maybe an IT or data ownership and management. So that way you can further scale them. That way you're not trying to engineer all your data AI assets first before you give them to the business. You said rely on the business to build them and then bring them back in and make them reusable. So kinda, again, tying to our overall themes around data, data management foundations, and architecture. Right? We've talked a lot about how do you build an ecosystem where you can now see the activity that's happening, you can see adoption. You have the ability to define and classify things. So now that you have that, you have this idea of reuse, which is a very popular topic among, I think, Dataiku customers, but also just enterprises in general trying to do self-service. Because if you can imagine, if I'm, in a self-service model and I'm a financial leader, I asked 10 of my analysts to go build me a financial KPI. It's very likely that those 10 analysts will go out and build the same name KPI, but to find it 10 different ways. And that starts to become a challenge because which one is the right one to use. Right? So kinda tying to our previous slide, this idea of what is the most used out of those 10 models, one will probably lies to the top and get most of the utilization because the business finds that to be the most valuable. And that's where reuse comes in. You wanna highlight that particular model out of 10 and then try to drive the organization to utilize that model versus trying to reinvent the wheel again. And that's kinda where this concept comes in. So as we look at that, there's a lot of reusable artifacts or elements of data within these type of environments. From a Dataiku perspective, your projects or your data products can become I think I got another question here. I'll I'll pause before I come back here. So from Ayesha. So each set of data model, do you have a AI bot example, finance, PA, RA? Yeah. Yeah. So I think, again, it depends. So if we're thinking about doing an agent, right, you've already kinda started to break things out between finance, PA, RA, like this different business divisions. That's kinda the idea of defining a model per division or doing that organized data I mentioned earlier. As you start to define what those look like, then the agent knows to use that corpus of information because you can say, hey. This is my financial data. This is my receivables data for the whole company. Now when you have the agent operate and you want to automate a receivables process, it knows to go look at that data as a master for it. So if you think about it in that construct, you could probably have an AI bot example for almost any division, almost any process. Right? Everything from, say, like, accounting. Accounting is very rule based. So you could have a bot derive certain answers from an accounting, we'll say statement or model based on the historical data you have and the decisions you would make to drive that accounting. Could do the same thing with an illegal. You can make those decisions with a a supply chain shop where where you need to decide if a certain tolerance is effective or not to move forward. AI can easily evaluate that tolerance and then make that decision again based off your historical data and the tolerances you've accepted before. So I think from that kind of perspective, you could probably take a AI bot or an agent and apply that to almost any field or space because what you're really looking for is replicating the historical decision making, and that really comes from pulling the data from that specific part of your business and looking at how you made decisions in the past. So hopefully that helped, Aisha on the question there. So let's come back here to to driving reuse for a minute. If you look at the types of reusable artifacts, as mentioned, you know, Dataiku itself can have projects and or data products within it that are reusable for the organization. Right? Similarly, your LLM models, your AI agents. So we've talked a little bit in the past about LLM mesh within our first webinar, and that's where that really comes into play. Right? You can use LLM mesh and say, hey. I wanna reuse these specific models. Are there the ones we've built internally? Are there certain external models we've approved? And I can direct my traffic to reusing those models versus using other ones that maybe I don't wanna use. So in that way, you're actually enforcing that reuse paradigm directly there. Another option is maybe your pipelines. So if you're in Dataiku, you could take snapshots of different parts of the flow and reuse those across multiple projects. That is a great way to say drive standardization in your development. You can reuse visual or no code models. So let's say you build a series of data science models over a number of years by your professional data scientists, but now you want the broader business to use them. You can take those models, the Python that runs them or r, and take that and load that directly in the data I could make them drag and drop and reusable. So now you're empowering using your own IP of your own models throughout the rest of the organization. Your workspaces. You can define reusable workspaces for users to make sure the same types of setups as they join. Your datasets. So if you develop a dataset in Dataiku, you can highlight that for the data catalog and make it reusable. And similarly, if you develop macros in Dataiku, you can make reusable processes. Again, a lot of these extend to the broader data ecosystem. We kinda talked about data products, l m models, datasets, etcetera already. But similarly, if you're developing visualizations, you can look to make those reusable. I'll talk with a number of clients here since, being a data at Google. We've had clients who have thousands and thousands of data visualizations, but maybe only 30% of those are being used. There's an exercise here to say, hey. If we're not using all of these, maybe we can drive more reuse and drive that focus. Right? That really highlights the need to put a governance layer of observability across the work individuals are doing. So, Marley, you can have reusable queries within your data lake ecosystem. So if you have certain queries that work well within your lake or mesh, you can provide those to users. Even things like your data move movement or data storage. You can automate ETL processes. You can automate the ability to actually deploy new instances of data stores and or data technologies. The list goes on and on. But the goal is to drive more and more of that reuse. Right? I think I have another question. I got a few questions over here. Alright. We'll go through those. So I'll go with, I think, Byron again. So you said Python or r r sorry. Just r. Python or r. So the general experience I see, folks do use still R, but a very common experience is using Python. If you have a, you know, model that you've created in Python, it is very easy to pick that up, plug that in the Dataiku, and then actually make that a reusable model, whether that's within the no code kind of model testing or potentially even make it drag and drop within the pallet. I have something with Python. You can also do some of that similar work with r, although, family, it's been Python on the market. I'll go back up to the top. So Renata again. Is there a sleeping bot and wake bot still not consume resources? Ah, yes. So, again, I think this comes down to when do you decide to to trigger a bot. Right? It comes down to the logic when you need to run it. So think about it like this. If I'm gonna run a bot on a schedule and I tell it to run every minute, then I'm gonna consume a lot of resources even within an hour because the bots run every minute for an hour. Now if I change that and said, hey. I want this bot to run, but only when this particular event happens, this particular set of data changes, some type of trigger. Right? Because I know when that trigger happens, the bot has something new to look at. A simple example of that might be someone just prompting the bot. But if you think about true automation, it could be, hey. We just got a new receivable item in our in supply chain inventory. Now that we have this item, let's trigger this agent to drive the next steps and processes. If you did that and you only receive a new receivable, say, once every, hour, now you've gone from running the same process 60 times in an hour to running it once in an hour because of the trigger. So to me, I think it kinda comes down to the idea of when you trigger that agent to run, whether that's a prompt or something else, and how do you do that. So there may be times we want to run constantly because you need up to date data. Fine. Then you won't kind of manage the cost to go into that. But I think in most cases, it's about how you architect, when does that bot run, and then also how much data does it use. Let's say you're in Dataiku, you can actually cache some of these results. So if you were to cache results, you may even use less processing each time you run it. There's a lot of ways to think about kind of reducing that overload, if you will. Okay. We got lots of great questions over here. I'm gonna go to Koichi here. This is less of a data question, more of a people question. How let me expand this. How do you effectively squelch the non best datasets that the creator might believe is better for him than what's deemed best? Tell you what, I mean, I'll come back to this slide. Sorry for folks who may be watching here, but I will jump up a bit. So it's a little hard to see probably on the slide at the bottom. But my usual recommendation is this idea of what I would call a published dataset council. You could also call it a reusable dataset council. So this is the idea of actually having your organization talking across boundaries or talking across divisions to talk about what is the right use datasets for the company. So in this case, this is kinda drawing from my own personal experience. What we would do is have a published dataset council where you have representation from each division of the company. I think ideally those are more from the business side and understand the context of the datasets they're working with. But let's say you have two representatives from each division. Right? And then what you do is you look to track the metadata, the consumption of datasets over time. And once you see a broader, we'll say consumption, you can pick a trigger. I'm also say 50 unique users in a month. Then you trigger the idea of bringing this data set in, having a conversation on the council saying, hey. We're seeing a lot of activity here. Let's talk about is this the best use data for the company? Now let's say that data set comes from supply chain. Maybe the two supply chain representatives can speak on that and say, hey. We think this is right or maybe we don't. Similarly, you have other divisions who can also chime in on that and potentially data professionals. But in my experience, the goal is to use data to drive the initial decision and then use conversation and use expertise among team members. Right? That subject matter expertise we talked about in the first webinar to actually use that to help drive an ongoing decision making, but what is the right item or not. I find if you have that kind of conversation, you can more move away from someone having a preferred dataset and move more of this model. Okay. We haven't agreed to option. And I think those conversations can be heated, but I think it's better you have that versus having split decisions around what is the right dataset versus what is not. Great question, though. Okay. So with that in mind, I'll come back to our flow here a little bit. So we've kinda talked about these types of reusable artifacts. So the next thing I'll kinda jump to here is the idea of how do you encourage reuse. I get this question all the time. How do I drive more reuse in the organization? Then it's hard. Right? I think most people, when they start to work on something, initially think I've got a new idea. And it's it's hard to kinda change that behavior to say, why don't you check to make sure that this has not been done before? So there's a few ways to think about that. First is training. If you have a training program, I recommend talking about reuse consistently throughout the entire training. Talk about the beginning, talk about it throughout, remind people as part of your user journey to always go check to see if this has been done. The more you say something, the more likely someone will actually hear it and adopt it and listen to it, and I think it's important to reinforce that. Another one we've talked about a few times today is the idea of measurement. I think measuring holistically, adoption, looking at things like that is a very strong way to drive those conversations around reuse because it tells you empirically, hey. This is probably a good idea because people are again voting with their clicks or looking at the things that make the most sense to them. I think having a decision making process. So going to Koichi's question a minute ago, what is the process for deciding when something is reusable or not? Right? Is it a internal team decision to the data team, which I wouldn't recommend? Is it a broader business decision with, you know, stakeholders and SMEs? You know, know, how do you actually wanna make that decision? Again, how do you communicate it? So with any self-service program, communication is absolutely essential, and that's actually something we'll talk about, in our upcoming sessions here. But importantly, for reuse, you have to communicate what are the right reusable objects in the organization. You have to highlight those. If you don't, folks may not know to even think about that reuse. So make sure that's getting out there. And then finally, again, gamifying things. Right? Especially gamification around the creation of metadata allows for more that reuse to happen because as you highlight, you know, make people excited for the gaming element, maybe through competition. At the same time, you collect more of that metadata, and that actually drives more reuse because there's more context and business understanding. And then finally, I know we've got another question here, which I'll jump to in one minute. But just to kinda go to the right side here on where to share, pretty similar to some of the things you've already seen, but the idea of, again, the AIQ data catalog, you could share directly there around your reusable assets. That's especially powerful. That's where users are also developing because you can kinda push them towards, hey. This is the right data. Let's look here first. Right? Similarly, Wiki can be used very much the same way. Enterprise data catalogs, very similar. Again, organized data and AI assets. Again, more we're going with multiple context protocol. How you organize easy access to these assets can really accelerate the use of them. And then finally, I think visualizing the reuse you're doing. Visualizing what are the most reused datasets, most reused reports. It can help drive more of that conversation. So we'll jump back to questions here for a minute from Samuel. Working in a super regulated industry, insurance, data quality is considered a is a cornerstone to all this framework. Would you have by any chance of link paper content addressing data quality state of art? So what I'll tell you is, you know, today, obviously, we're focusing on data management, which definitely is very close to data quality. What I'd say is is, really over the next two sessions, we'll talk more about that. Because really, it kinda comes down to both a people perspective as well as a a governance perspective. Right? And our next sessions will focus more on what are the people roles and operations that go into this. And then webinar four will focus more on that governance around it. That said, what I would tell you is is a general high level, is that I believe data quality is best served through the actual, you know, we'll call active use of data by a self-service community and then feeding that back in to improve quality. So something I see happen a lot, especially with traditional data engineering architecture team, similar to where I came from in my career, is this idea that, hey. We're gonna engineer all the data up for it. We're gonna get the quality exactly right and then give it to the users. The usual problem with that is it could take three, four, five years to get that done. It takes a lot of time to get agreement and engineer every dataset. So what I usually recommend instead is kinda we shared a couple slides back, is this idea of organically capturing those best use datasets and then feeding those back in right to be reusable. Well, quality follows a similar approach. If I think about a self-service user working on, we'll save base layer data within a data lake, they're gonna find those quality issues in the work they're doing because they're gonna be trying to solve something. They'll have some expertise around those datasets because that's the business they work in every day. And when they see, oh, my quality is not right here, that's a really good idea to say, hey. Maybe we address those quality specifically because it's impacting an actual project of work happening at this time. And the best practice I usually recommend is if you think about having a a quality submission function. Right? So let's say you, you have a data owner of an ERP, and there's been some quality issues found in the ERP. Right? The self-service users can then add their quality request to, maybe, say, a change log for that ERP. You could even have users vote on what changes to make, and then it becomes a data owner's responsibility in ERP to make those quality changes. Once they do, that flows downstream to the data lake or mesh, and then multiple teams can then consume from that and get better quality data. And what's different in that model is that your quality is being driven by what the business is seeing and needs on a day to day basis. It's tactical and close to the work versus trying to predefine all the quality upfront and really trying to boil the ocean. I'm sure we'll get more of that in following sessions, but that's how I describe it at a high level. It's more of a focus on quality for the specific need that you have versus quality for everything. Great question, though. Okay. So I'm gonna keep this moving here. Kinda last slide we'll cover here for today, and then we'll be open for any other q and a. So the last thing that would tie to this is the the concept of kinda data product ownership and retention. Right? So let's take this idea that we have now got a large growing self-service community. We've built a lot of new, you know, analytics, models. We've got agents coming from the business. We got all this work going on. Right? Well, one of the challenges is that, more people change jobs. They lead companies. They change roles. Right? And you have to be really kinda smart at how you manage the overall footprint, have everything happening in self-service, all the jobs and processes to make sure that you're not running things that aren't really driving business value, and also to make sure that you don't have, a lot of things running that have no ownership. So if something breaks, there's really no one there to fix it. Right? So as we think about that, the first thing to think about it, what are the things that you can manage? And this list can get pretty long. We've talked about a lot of these today. So products, projects, so agents, a couple of new ones I'd add here, like your metadata scanning. So you're scanning related to maybe your enterprise data tool, scanning your environment for metadata. That's actually a process too. Right? Your ingestion jobs, the jobs that bring in new data on maybe daily or weekly basis, those are also processes that consume things. So there's lots of things that happen on a regularly scheduled basis. Right? Additionally, you also have some items around ease of use. If you have too many of these and they're all said to be reusable, then it's actually harder to use something because there's too many options to make. There's too many selections. Right? And that's where I put things like reusable pipelines, you know, reusable visual and no code models. Those don't necessarily use resources on a regular scheduled basis, but they do make it harder to make decisions. So if we wanna make sure we're controlling all this and that we don't have, you know, thousands of things running that maybe people aren't using, there's really two things you have to look at. First is making sure that you have ongoing ownership of those data assets or data products. Right? If you don't have ownership and you have an operational incident, someone may not even know how to go or solve something because there's no one there to go to. Right? So it's important to have someone actually understand that's been built. Ongoing viability. I would argue that if there's not an owner of something, it's likely not gonna have long term viability. So you need to question, do I even keep this around? If you don't have an owner, you also lose your SME knowledge. An underpinning idea of the series is that the business understands the context of the problems they're trying to solve the most. So because of that, you have to make sure that you have SMEs from the business to understand those processes. Because if they're not there, it could be really hard to maintain. And then fine, there's also some issues around increased retention issues. Right? How do I make sure I'm retaining these things over time if I don't have an owner to make sure I'm doing that? So a couple ways to look at it. First and foremost is automated. I always recommend, you know, look at your HR systems, make sure you know who the owners are of all of your products. And then if someone changes, they change in the HR system, they change roles, leave the organization, make sure it's a process to either reassign through a manual process, from the employee or potentially from the manager. But make sure there's a way to enforce that there's ongoing ownership of this. Right? And then similarly, there's a business process approach. So, again, training, communication, documentation. I'd recommend re certifying ownership probably on at least a three to six month basis depending on your business. And if there is no, owner to certify, make sure the managers are certified. Make sure you have a belly button tied to the asset. And then retention, which is pretty close to this, but kind of relies on ownership. Retention is the idea. Do I continue to run something or do I remove it? So as we've talked about a lot, waste and increased operating costs, so even going to the idea of a sleeping AI agent versus not sleeping agent from a question earlier. So similar idea. Right? Do we need to really run all these things all the time, or do we only need to run them when they need to be ran? Or maybe we just keep the metadata, but don't run them until they're needed. Right? There's a lot of ways to think about that. But the idea is that if you keep running everything all the time, your cost will keep going up, and you may not get all the benefit from that. Similarly, there's, again, reduced ease of use. Keep too many things. It's hard to actually stay focused. And then tying another one in here again, there's residual compliance risk. Every data process or project you run, does have some form of risk tied to it just by the nature that you're working with data that you're relying on it. So if you don't need something, it makes a lot of sense to turn it off just to remove the risk entirely associated with what that could be. So, again, from an automated approach perspective, track your own utilization. I I can't say this enough. If people are not using something, it's not bringing value. And if people are clicking on it, it likely is bringing value. So utilization and adoption is very, very key because that tells you what matters and what doesn't, what you retain. Right? I think it's good to look at artifacts that haven't been used for a period of time, so say six months, and then actively reach out to those owners to make sure they certify something. You do not wanna have a bunch of reports running at two years old that no one uses. It's a headache for other folks to clean up later and eat your resources. So look to make that part of the culture of the business to always consistently look at that. And then, yeah, if you are running something, you don't need it, either turn off the schedule or fully remove from the system. Right? Make sure you're keeping a clean house. From a business process approach, this is very identical to looking at it from the ownership perspective. Trade and communicate, make sure you document, make sure you recertify the viability on a regular basis. Is this still driving value? And in the absence of owner, make sure you go to a manager if there's no response. So with that in mind, I think we'll transfer some of the questions here. I think we've got about twelve minutes. I've already got one other question in here, but if folks have additional ones, please make sure to to go ahead and add them in. So I believe this is from, Tilla. How can we monetize given value to these issues that CFO understands? Sure. So I think through a lot of things, right, the best way to get understanding is to actually make it real. Right? So if you're the CFO of a company, right, and you are relying on your financial numbers to make a decision. Right? You're reporting to Wall Street. You're making decisions about where the strategy of the company is going to go. A good example is to say, okay. This individual needs this information. Let's show him what we currently have. We'll try to build the report, but let's highlight where the quality issues are. Right? So let's we can even remove AI from this conversation. Just say it's a simple visualization. If you build that visualization for that CFO, but you highlight, hey. We have some confidence here, but not as much as we'd like because there's a quality issue. What do you think about now that you've seen the structure of this, what do you think about us pursuing improving the quality of this and working with the business or say the ERP owner to do that? I think when you make something real, when you attach to what someone tangibly knows, that's the best way to get interest in that. So that's one way. Right? I think another way and this is something we'll talk about probably in session on webinar six, right, which is around value representing value. I think another way to do this is simply to show the value that happens from, you know, self-service or empowering the business in general. Think about, like, a simple roadshow by about having the ability to show to maybe senior leaders of different divisions how the value is coming in. I'll give you one example from my past. Early on a program that I was leading and working on, we, we're really doing a grassroots approach. And then we actually had a session where six of, I'll say six of younger engineers or young professional engineers in the organization shared examples of some of the major outcomes they had driven using no data accurate or using self-service. Right? And through that, they're gonna show immense value to the SVP of aerospace engineering who have been leading that entire division or stack. Right? At the close of that, that same leader then decided to commit an extra million dollars into that program to help them grow. As I like to say, it went from grassroots to kinda top down on an overnight basis. And the reason that leader did that is because he understood what his engineers were going through. He understood they had a lot of trouble getting the data. They're spending a lot of time working with that data. So we're able to show empirically through examples given by his team members, here's how they are helped and driven forward. That made that real for them. So I think at the end of the day, it's always about starting, looking at the grassroots function of the thing you need to solve, and then highlighting the value that comes from that or the value that could come from it in order to help influence that decision making. Okay. So I'll jump to the next one. We got a question from Alex. So apologies if covered, but curious on your thoughts on data structuring. Have heard vector databases, graph, hybrid, no need to do anything given upcoming connectors and increasing model context, windowed, if 99% unstructured data. Yeah. Yeah. So to your point, I think it actually comes down to the intended use and use case. Right? So as an example, in this session, we're directly talking about the idea of empowering individuals to use self-service. We're also talking about the idea of AI and agents utilizing data. Now, fundamentally, the tagging or metadata around that helps both those groups. Right? So regardless if you're in a vector or a graph or a relational database. So to me, it actually comes down to more what is the intent of what you're trying to do with those datasets. And if you're looking to do certain things, a vector database may make it more sense here. A graph database may make more sense here. Right? It really comes down to that idea and same for structure or unstructuring data. Unstructured data sets are gonna be pretty hard for someone in the business, say, a sales and marketing team member to pick up and use. Right? That's gonna be difficult from a self-service perspective. However, if you're gonna just run an agent against it, then it may not matter because that agent can look through and rationalize that data whether it's unstructured or not. So to me, and I think a lot of your question comes down to what your goals are or where you wanna go. The one thing I would say that resonates throughout, though, is you need to have a performant architecture that scales that specific need. Right? Something that won't get in the way of their users or what your agents are trying to do. And you need to make sure you have a good understanding of the underlying kind of data management and data that you have that you can guide both of those effectively. Right? Even within those graph and vector databases, you're still gonna need that metadata management and really that tagging to drive that decision making for an agent. A really good question. Let's see. We got about eight minutes left here. Any other questions here on the oh, we got another one coming in. Great. Love the questions in these sessions. My favorite part. The recommendations you provided by questions to ask are not only of great value for COA AI agents, but we're general as a foundation for every company that wants an efficient, effectively created value from data with or without AI agents. Agents give us now a good line of augmentation to push it through. Right? Yeah. Great question. Now you're kinda hitting on this as well. As I've kinda mentioned starting the series, I've looked at this for about ten years now. Do I do tend to agree with you. Right? I think the future of how organizations succeed largely comes down to how they manage and work with data analytics and AI at scale. And a lot of that comes down to the foundation of how you look at enterprise data, how do you enable the organization to work with data. Right? If the nature of our jobs are changing, what we do is becoming more data centric, we have to build that capability. And my belief is is that we as we move more into a world of AI and agents, that's only gonna accelerate even more. Right? Because agents really make things go faster. It's not just an individual going through and doing research in every step. I can ask that agent to go out there and do all these steps very, very quickly and get to my answer. But interestingly, I still have to be able to point the agent in the right direction. Right? If I'm if I'm doing, like, guiding a ship, yeah, I tell the ship, hey. I wanna go across the Atlantic Ocean, but then I pointed north and I go to Antarctica. Doesn't matter how fast the ship went. I still ended up in Antarctica. The same thing kinda applies when you're looking at AI and agents. Right? If I don't have the right direction, metadata, architecture to do all this, it's great that I can run an agent. I can spend money to do that. But if I want the right things to get the right answer, I'm not gonna get there. And I think because of that acceleration and because we all know that that automation allows our businesses to scale faster to be more successful, it's gonna really highlight the need to have this foundation. Because if you don't have it, you can't really effectively leverage AI. And if you can't leverage AI, how's your business gonna succeed against other businesses who are leveraging AI? So in my mind, all of this happening on the world of AI agents, where we're going, this automation that's occurring, just really critically highlights the need to continue to focus on these items even more so than they were focused in the past. But very good question, and thank you for also for the kind comments. Great. Just a few more minutes left here. Anything else on the on the call for questions? Really appreciate them today. They're great. Any more questions, anyone? Oh. Oh. Got it all. Alright. Oh, this is well, see. I'll let Renata take one from Samuel. Yeah. I'll do one from Macca real quick, and then we'll switch to nontechnical. So does the ETL model work with agents? Yes. So, again, it depends on what your use case is, and this is why the idea of a mesh versus, say, using a physical environment and doing ETL or what I usually call ELT. Right? You extract it and just straight up load it. Don't transform it. But, yeah, I think the model still works. It just depends on how you wanna organize your data. Right? If you're strictly building agents and you said, hey. I want this agent to go to these different data resources and they're distributed across your business. As long as you have access for that agent and your users could see it through a data mesh, then, no, you don't need the ETL. You can still go out there and do all that work. But by the same token, if you were to ETL everything centrally, it may be easier to go find that and maybe easier for the agent to access it. So I think you can do ETL one way or the other. To me, it comes down to how do you actually wanna manage your data management organization. Do you wanna be more physically kind of centralized throughout the company? Do you wanna have more of this distributed capability with different teams building things connected over a mesh? The important construct though is if you have a enterprise view of all the datasets and where they are. ETL can play a role in that depending when you need to move data or not. I do believe in general though that we're more leaning more towards virtualization, and that we'll see in the future a lot of AI working through virtualization to access datasets. And I think users will do that as well. Cool. Great question. I think, Samuel's question was answered by Renata. So just to confirm for folks, I'd like to check the prior session and the coming one. Renata has actually shared that directly here in the chat. I don't know if you'd like to go look at this in the q And since I see no more questions, I think I'll pass it to Renata. But, thank you all for the time today. We'll be kicking off the next two, sections of this webinar on the upcoming month, so look for those updates. As mentioned, webinar three will focus on more of the people, operations, and kind of funding that goes into things with the operating model. The webinar full of that deep into things like governance and architectural governance and how we build that framework, least type of programs. But with that, I'll pass it to Renata. Yes. Thank you so much, everyone. Webinar three in the series will be coming your way sometime in June. So, as Jonathan mentioned, we'll be diving into the topic of operationalizing self-service data with a focus on people products and funding models. We'll be sharing more details soon, so please keep an eye out, on your inbox for the invite. And as always, we'll be sending out the recording for this session's slides, from today. And thanks again for joining us. Jonathan, big thank you for another great session, and see you all next time. Thanks, Ron. Have a great one, and hope to see you soon. Bye bye.