Video: Introducing Dataiku Agent Management: See it live | Duration: 3440s | Summary: Introducing Dataiku Agent Management: See it live | Chapters: Welcome and Introductions (4.56s), Agent Proliferation Challenge (79.555s), Customer Discovery Process (262.65s), Agent Management Overview (485.875s), Agent Discovery (613.76s), Agent Monitoring (765.465s), Agent ROI Metrics (902.215s), Agent Risk Management (961.125s), Market Reality Check (1136.69s), Agent Value Measurement (1198.245s), Live Demo Introduction (1346.52s), Agent Management Overview (1442.68s), Cost and Risk Management (1594.985s), Agent Infrastructure Setup (1691.01s), Critical Safety Investigations (1771.44s), Agent Monitoring Metrics (1809.795s), Agent Observability Tools (1915.56s), Advanced Agent Features (2143.325s), Q&A and Wrap-Up (2283.345s), Dual Ownership Model (2407.28s), Alert Rules Configuration (2531.485s), Agent Topic Generation (2680.865s), Security and Gateway Features (2851.55s), Architecture Overview (3094.055s), Common Mistakes & Lessons (3202.71s), Closing Remarks (3400.275s)
Transcript for "Introducing Dataiku Agent Management: See it live": Hello, everybody. Welcome to today's session. Some folks jumping in. Today's session is introducing Dataiku agent management, see it live. Thrilled to be here. My name is Austin Cook. I lead the solution engineering and customer experience team for The Americas, and I'm joined by Jed. Hi. Jed Doherty, SVP of AI and platform for Dataiku and, one of the core developers of agent management. Yes. Very exciting. So thank you all for joining today. A few housekeeping items. And so first and foremost, thank you for joining. Secondly, please do, leverage our q and a and chat, which you can see hopefully on the right hand side of your screen. For those that are interested in the materials, those will be shared, with you, after the event as well as the recording. So with that, let's go ahead and jump into it. So last year, the building of agents was really a a challenge for folks. That it was a newish technology. A lot of folks were experimenting. But, ultimately, what we found is, that that's no longer the case. A lot of people are building through, you know, their own vibe coding tool of choice. Certainly, a lot of... Hopefully, a lot of folks on the, call here are building and built many agents inside of Dataiku or other supporting technologies. On top of that, I have to believe several of you are experiencing, existing software as a service companies or SaaS vendors that your company are leveraging are are introducing their own agents into those frameworks as well. And so the number of agents, out there is continuing to grow. You can see some of the facts and figures here, where, by 2027, the number of agents is expected to grow 10 x. And, naturally, as more agents come into the fold, whether they're ones that you're building or ones that are, injected through, your own, software apps... Applications or even, point solutions that are being brought in, with or without, full enterprise IT approval, agents are here now. And so, naturally, as these start to grow exponentially, you're starting to see, that last figure on the bottom where really less than a quarter of all agents or or all organizations actually keep a complete current AI inventory. And so, as you can imagine, a lot of people liken these agents to employees, and so you can imagine the, challenges and some of the risk associated with having employees or agents that you're unaware of. As a addendum onto that, what I've often found when I'm speaking with executives, especially if there's, like, a couple executives in the room, is, you know, awesome. But this but this challenge, like, oh, do you do you know how many agents you have? Do you know where they are? And one, you know, director of data science would be like, oh, yeah. We all build agents in tool x, and we've deployed 16. So, yeah, our company has 16. And then somebody else in the room, director of AI or the chief information officer might pipe up and say, oh, you're building on platform x? We're building agents over on platform y. So we see people discovering in conversation with them, that they've made multiple different decisions. So it's definitely... Even if your team is all aligned, like, it's very unlikely that the company as a whole is aligned. Yeah. Exactly. And and I think that's part of the the challenges you said, Jed. It's it's... Nobody is unified. It's a very complex ecosystem currently, which is an opportunity in in our eyes. So when we're having conversations and and, you know, candidly, as Jed and and our product team are aware, like, we at Dataiku, particularly myself and a few folks, had customers coming to us with this problem, really back in, like, November or October, November of last year saying, hey. You know, Dataiku, I know you guys build agents. We love you for that. We know you do you do governance quite well. We have this problem where we we have all these agents popping up, and we're really, really scared. And so, you know, we had a lot of these conversations. It was great. And I I funneled that, feedback to our product team, and we got them in the right rooms. And, a lot of, you know, those conversations led to us to make this decision to to ultimately move forward with the product. And some of these questions that happened in those early calls, you'll see here on the screen. And so as as Jen alluded to, how many agents are we running? Depending on who you ask, you might get very different, answers to that. You know, enterprise IT may say, well, if I look at my AI registry where everyone must come to me to get approval to build an agent, it's maybe 20. But if you go talk to, a head of data science, they may say, oh, yeah. We've got 15 or we've got 40. And then if you go to the actual employees, maybe people that are building things, without IT knowledge, it may be, even more. And so a common question that, unfortunately, many organizations can't answer is, how many agents are we running today? And, obviously, you know, there's a lot of conversations, a lot of materials on just the the rising cost of tokens as it relates to to agents. You know, we all thought Jenny and I was a lot, but agents, tend to exacerbate that issue. But I think in general, the the the check is coming to to get cashed, and a lot of folks are struggling with, hey. We've we've had this top down mandate. Go build agents. We've done that. Now we're getting the bills, and we're saying, well, actually, is is this really worth it? And it can be quite challenging to actually understand the ROI of those agents. And then, naturally, these agents... Because, you know, the more and more they get involved in processes, they more and more they get involved in decision making, there's inherent risk to that. And so, it's a very challenging, problem to understand across your entire agent landscape, even the ones that you know about. But what what risks are they introducing to the actual, to the business themselves? And so common things around like, well, you know, are are we are we siphoning out, data? Are we sharing IP? Is there a a regulatory risk? Things like that. It's very challenging to understand that across an entire agent landscape because, again, not, everyone who is building agents are familiar with risk profiles. They're not familiar with, various risks across, these agents, and so it's just challenging to catch up. And the last one there really dovetails into what we're talking about earlier is, yeah, business impact is important. It's certainly part of the equation, but even just understanding the costs. And, yeah, if, I... You know, my one and only, LLM bill, it it comes through. I'll sort of know the cost, but can I get that down to the agent granularity? Can I get that down to the infrastructure provider? Can I get that down to the line of business? Having that level of visibility is what a lot of, organizations are after. So all of that is why we developed Dataiku e and agent management. And, really excited here to share more about, agent management, what it is, and why we think it's important for for many of organizations. And it really comes down to three pillars related to what we just talked about, Discover, monitor, and control your entire agent estate. And I will emphasize, entire agent estate. And so it's really focused on not just data accuagens. Of course, again, many in the line here probably understand and are familiar with Dataiku agents, but, we're taking our agnosticism to tech to the agent world. And so we're focused on not just your Dataiku agents, but across a multitude of of agent frameworks, which we'll touch on here, quite shortly. But, again, part of this process is just discovering what's out there. So we will connect, to all of your agent, development frameworks. We'll ingest, we'll monitor, we'll show you what your actual agent landscape is across the entire, landscape to create one true inventory. From there, we're gonna monitor and understand the agent value. So tracking, of course, things like health usage costs, and and those aspects. There's a lot of vendors, that are focused on just, you know, pure observability stays. But what we found is if you really want to manage and treat your agents as as humans or as true ROI, you need to understand the full business value as well. So what KPIs are tied to the success of that agent? And, are you continuing to see the return on your investments? The last side of this is controlling, so controlling against risk. Being able to understand across my entire agent landscape, where am I exposed? Where are my risks? How do I get alerts to understand when I need to take action and actually course correct with a particular agent? So let's dive into a little bit detail on that. So first on the discovery side, every agent from every platform, all in one inventory, all in one catalog. That's what we're focused on. So, it's very important for us, as I, again, have had conversations with folks, they say, hey. Hey. Yeah. We have an AI council, and and, you know, everything comes through us. And that's probably largely true. But, again, there's probably some Venn diagram. There's some outside space where perhaps people are experimenting, people are doing things under your purview, or they're trying to prove out, if it's worth it before they actually come to your counsel. So we're trying to solve that through this automatic agent discovery. The first step in Dataiku agent management is defining your agentic platforms that you'll scan and register to create that, unified ecosystem. So you can see, on the lower side of that screen all the agent frameworks that we support, AWS, Bedrock, AgentCore, data Databricks, Datacu, Google, Microsoft, Salesforce, Snowflake, as well as, coded agents through OpenTelemetry. And this is just where we're starting. Obviously, we're having a lot of conversations with customers. They're bringing more requirements and more frameworks that, that we will likely incorporate into the road map, down the line. Along with those agents, we'll also do some metadata enrichment, which helps us, again, understand and map agents to business processes, where they sit in your organization, where, enriching a bit more context with the agent. So it's not just a chunk of code. It's not just, an an API, but it's actually fitting into your business and your, and your organization. And, of course, having this... Just having this catalog, this one stop shop allows you to search across that vast landscape. And so think about identifying where you might have duplicative agents, where, you know, dev teams aren't talking to each other and trying to solve the same problem. Naturally, that's a great way to to burn tokens and, cause confusion. And so having that unified view allows you to have just a a better, understanding of your landscape and stamp out where, you know, potentially some shadow AI as well as some duplication efforts. Monitoring is core to this. So once I have my entire inventory, I wanna see what agents are working and if it's worth it. And so on the top side, you see it's really focused on the performance monitoring, which again is sort of core to any sort of API type framework. So is it operational? Is it up? Wanna make sure that, if it's, important to the business that it's it's actually operational. That's, it's up. It's not, reducing latency. And then also, is it being used? Obviously, an agent that's not being used, not not the end of the world, but, again, if it's introducing risk, perhaps it's flying under my radar. And I wanna make sure that I'm tracking things like the, monthly active users, daily active users, sessions, chats, things like that, and then performance from the agent. So, think about your traditional sort of agent evaluations. On the behavior side of things, you know, there's a lot of agents that may look, healthy from a pure operational standpoint. Again, red... Green lights across the board. It... It's there. People are using it. It's great. But it's actually doing the things that I expect it to do, and that's where the behavior analysis comes in. Being able to detect changes in agents or, just as importantly, users' behavior, how they're interacting with your agent. So things like tool drift analysis. What are the common patterns for these tool calls when I tell it to, score a a, probably default model, is it actually doing that, or is it making up a score? That's important for me to understand. Topic modeling for usage analysis. I don't want, people asking my customer service agent to write Python code. That's, a not good use of of of my tokens. And so being able to understand topic modeling for, you know, guardrails that I might want to, build into my agent or even just for, being able to understand features and widening of scope or even refining scope to my agents. That's not something that, you know, uptime is going to tell you. ROI is critical for all analytics, machine learning, and the like. Agents are no different except, as we're all probably finding out together, the costs, can can be quite, expansive. And so being able to understand not just the cost, the tokens, the interactions, but also tying that in one unified view to show the business value, so the business value associated with those agents. What KPIs are important, and, really, where the justification for building an agent in the first place. Manual escalations are a common one. And so if we're manually escalating things, we're likely not getting that return on those tokens. It's It's going to happen, but, again, it's probably built into your justification for doing this agent that an escalation stays within within some threshold. And having that understanding is is critical. On the control side of things, every risk carries some... Or sorry. Every agent carries some risk. That is a fact. And so being able to understand and assess and certify those agents through, through one unified lens is critical. As I said earlier, a lot of times, the data science teams or even, you know, your your core developers, your end users building agents, they're not in line with the the risks of the the organization. They, you know, may be, unaware of GDPR and the EUI act and everything else that's, comes to bear. And so being able to create a unified risk assessment across your agent landscape becomes, especially important. And, again, being able to say across your entire landscape, here is my risk profile, and, certainly, like all things, there's some risk that you'll accept, especially if the ROI is there. But if not, then there's other risks that you'll, likely not want to, stand with. And so being able to understand the risks, mitigate those risks via testing. So building dedicated set of tests, and validating those agent behaviors and alerting when those things are are popping up. And so defining, detecting bad metrics or bad behaviors and being able to alert those developers to actually take a corrective stance and, mitigate those risks over time. Risks like these continue to evolve, and they'll change all depending on, the overall risk profile. So that hopefully gives you, a little bit about agent management. One of the things that we thought would be interesting is to just stop and see what, you and the audience think about these things. And so I'm gonna share a poll if I open it up and share. Hopefully, that pops up in your screen, and, I would love for the audience to chime in here. How confident are you that you know how many agents are running across your organization? This is sort of that that core first level question. I know it's Jed's favorite one as he jumps into things with customers. And it's one of those that we we love asking before and then, seeing how it differs once we actually, install and configure Dataiku major management. So the votes are flying in. I got a few that are very confident, which, kudos to you. But it looks like the majority are partially confident, which checks out. I feel like that's where I expected most to end up. Jed, what do you think? Yeah. No. It's pretty much pretty much what I figured. So this matches what's the reality that I see in the market right now for sure. It's it's, yeah, it's tough. It's tough right now. It's tough. Alright. So let's close, that out. tough, but it's not necessarily, like... It's not a bad thing. We're we're just figuring this out. So it... Yeah. It's okay that the enthusiasm for building agents and making them useful has outpaced the ability to track where they are. I think that... That's healthy. I... Yes. I I completely agree. I think, these conversations have affirmed you're in good company. You're not alone. Don't go panicking. But, yeah, this is this is a tough problem to tackle and, again, one that we're excited to tackle with you. So now let's share our next poll. So we we kind of introduced data, Dataiku agent management in the landscape. Back to those four questions. So what do you think is hardest for your organization to answer today? So what agents exist? Kind of that core to that visibility. What agents are drifting beyond their approved purpose, what agents are delivering clear value, and what is each agent costing us. Let's see how the numbers roll in. So, yeah, some some core visibility that's expected. But, yeah, it seems like we've got another clear winner. What agents are delivering clear value? Which, again, I think, you know, Jed, you and I talked about this. Like, it's one thing to see the the denominator, the cost, and I think a lot of people are hyper focused on that. But being able to actually tie that to the business KPIs and the business value is one that I know many people are are chasing. So that that checks out. And to some extent, even with data acquisition management, you're not getting... There's not a complete magic bullet for this answer. Wish there was. But there's still some element here of... It's certainly useful to know where all your agents are and what they're doing and how people are using them, but then you still need to, as a business, think about what success looks like for an agent and be able to define that. No no tool can do that for you. I think a lot of people build a lot of agents for fun. I know I do. Without a clear idea of, like, okay, what would what would good ROI look like? What would my metric be? There still has to be that level of thinking. Yep. Totally agree. I wish they were a magic bullet as well, but, this is where, you know, these are different personas. These... Getting back to our developer standpoint, like, sometimes the developers don't even understand the business outcome that's being tracked. And so being able to create that unified view is is something that I think is, there's thirst in that for the market. Alright. So, I will quickly go back here and share and introduce our demo. So probably why you're all here, because that was the anchor of this session is see it live. I promise you, you will see it live, and Jed's gonna take you through that. And so, here's a quick scenario. I don't know, Jed, if you wanna introduce this or if you want me to. Take it away. Grounding of what you're going to see. So we have a fictitious... I'll emphasize that. Fictitious. This is not based on any, any real world because we don't wanna share what we've been doing and what we've seen. But, again, just generalizations of, common conversations that we're seeing out there, but this is a CIO of of Terrastack. Jed's gonna be playing, Dana or at least a a lieutenant of Dana. But, this hopefully is representative of a lot of the folks on the call. There's a complex agent estate. Yes. There will... You will see some agents in Dataiku, but you'll see a lot outside of Dataiku as well. They're certainly getting value from agents. Not all, but they are getting value from agents, but, also, the risk is uneven. So some of these agents, certainly causing some challenges, and, we're gonna show you how we are going to address them and, with data acquisition management. So take it away, Jen. Thank you very much. Alrighty. Let's jump right in. Check the screen. Great. Everybody should be seeing my screen right now. So this is Dataiku Agent Management. Here, as mentioned, I'm, I'm not gonna try to be Dana, but I am Dana's assistant, second in command, to Dana here at this, this manufacturing company. And we can see here quickly that I run Dataiku agent management, and I have connected a pretty broad variety of different platforms to this agent or to this agent management system. So we can see that we have a distribution of agents across Dataiku, Databricks, Bedrock, Salesforce, Google, Snowflake, Foundry, Copilot, all of which will be available at GA. So these are the things that we are connected to currently. There's a couple others as well, but this is the main list. You can also, of course, connect the... Any code agent, using, the underlying, basically, OTEL capabilities. So I have this overall inventory here, and this inventory is not static. So as new agents show up or get added inside of any of these platforms, we automatically detect them and include them inside of this inventory. I'll talk about investigations a bit later, but we can also scroll down on the main page here, and I can understand a quick overview of the activities. So this is the number of interactions or conversations that the agents are... Have been doing over time. We see token burn, and we could also see cost across our entire estate. We'll dive into these, different categories in a minute here for a deeper in... Insight, but at a high level, this is what we're getting right out of the box on this home page. If I wanna get a deeper view of the full inventory of all of my agents, pop over here to agent inventory, and I can see information around usage over time, discovery dates. So this is basically when agent management found this given instance. And I could filter down by, for example, like, my marketing. Here's my marketing agents, or I could filter by, different teams or even different infrastructure types. So here's our fleet... Field engineering automation instance. These are actual real agents that are that we use internally, connected to some real data there, but we'll pretend it's Dana's data. Beyond just a list or an inventory that, as I mentioned, is auto updating across all the different platforms, I can pop into the cost and understand the global cost, of my agents as well as, of course, filtering on whatever underlying things that I want. So I can see I've been burning around $500 a month. It's actually pretty cheap, maybe not a big deal, but we can get those numbers up. We can see cost per day by platform, my most costly agents, and I can even see things like model breakdown, so cost by different models inside all of the different agents being used. So looks like my... By 63% of my total spend is going to CloudOpus four eight on Snowflake. Maybe I can try to optimize that and save myself some money. Beyond the cost, as we mentioned, or as Austin mentioned already, we have this concept of risk. So if we know what an agent is doing, generally, we can know what risk categories it's most susceptible to, which then gives us this great risk matrix and allows us to mark whether or not agents are, the risks, inherent in the actions of that agent have been mitigated or not. So if I drop in here, maybe I wanna look at this one. Potentially, we have a potential hallucination, prompt injection, and, harmful content generation. So we wanna make sure that, the implementation to deal with these things has been resolved. Last, kind of administrative, but, yes, you have your different agent infrastructures. Adding an agent infrastructure is pretty straightforward. I can choose my infrastructure type. We talked about that before. Say Salesforce, and then I can connect to a Salesforce connection that I have set up. So there's... I need some level of credentials to go and pull this information. I hope this works. Love doing things live. And now it's going out and identifying the agents, that it found on that remote infrastructure. Once I have agents found, I can set up, for example, auto log fetching. All of this is gonna be defaulted to intelligent things. But, yeah, log fetching, uptime tests, telemetry tracking, I could ignore them if I want, etcetera. So I can control exactly how I'm tracking and in what depth, all of the different agents that are being auto discovered. And I can also set up, default rules. So as more new agents are discovered, whatever subset of things I want to have happen to those agents will automatically happen. Alright. Great. So this is kind of my high level observability side. Right? But I don't just wanna observe. I wanna be able to take action. And, ideally, I'd wanna be able to be prompted, to take action about the most important things, that I... I'm dealing with. That's why we have this concept of investigations. So here, we can see that, this is actually yesterday. This factory factory floor asset data agent, has been, flagged with a critical challenge. So I can pop open this up here. And we could see that it passed a critical threshold, of... Basically, it's offering unsafe operating advice. This is a, factory floor agent that is supposed to be providing advice about how to fix machines on a factory floor, and has been providing unsafe advice 18% of the time. How do we figure that out? Like, that's a really specific piece of information that, you know, obviously very important. I wanna be able to do something about it. But it's... It... How did it get to that point? So let's dive in here and understand that. So I'll pop into this agent, and you could see that for every single agent, we're really tracking it across three high level categories. We're tracking operational metrics, we're tracking usage metrics, and then we're tracking business metrics. And, really, all three need to be working effectively, for, you know, an agent to be deemed fully healthy. So here we see operational, got a warning. We'll take a look at that. For usage, usage is healthy, steady, and then we have saw a critical issue with the business here. So let's look at this. If I start with operational, I could see that we have my input output tokens. Looks like my input tokens have dropped below a threshold, recently, so that's potentially a little weird. Maybe I need to look and make sure that, the usage has not shifted. But this is sort of the... Kind of the absolute base level API type stuff that, you know, we've been tracking for years with any type of, well, API. Right? Any model uptime, for, like, machine learning model, we would be tracking these same things, latency, token usage, interactions, etcetera. So this is great, but it's really table stakes. Similarly, you know, cost, of course, we need to keep track of the cost of every agent, obviously. But, again, you could do that inside any individual platform and do some other type of aggregation. I don't think this is necessarily the operational, and the cost part is not necessarily the hard part. It's a necessary part, but not the hardest. Usage, also pretty straightforward. Just tracking how many people are on it. But now we get to the more interesting stuff. So one aspect of the more interesting, capabilities that we have. Note that this is not a Dataiku agent. This is Snowflake agent. We were able to go out and read and fully identify this agent's structure. So we have the platform configuration here, but I'm also getting this view of the components, basically, the abilities that this orks... This agent has, code execution. It's able to do a lookup. It's able to hit a second orchestration model for incoming usage here. I can see the tools that are available to it. So one is code execution, and one is use this tool to look up the latest days factory asset data. Not very exciting. And any sub agents that might be, getting leveraged here. So I have this in-depth view of kind of the components, the nitty gritty bits, underlying the agent. For business metrics, and remember, this is the critical part, we could see that a lot of these business metrics, are kind of the... Are we getting value from this thing? So here we... Over the last thirty days, we've avoided, twenty point one eight hours of downtime across our entire plant. It was tracking lots of different, lots of different, assets inside of our factory. We just... We have saved $20,000. That's great news. Where... When it recommends actions based on risks that it, identifies, where at 82% of the time, people actually take action on those risks. So most of the time when it alerts you, it's worth doing something about, which is good. But what we see here is this metric, unsafe operating advice. So that was very, very low, up until recently when it spiked up. So what does that mean? What's the details of that? If I look here in the behavior tab, the behavior tab is tracking the topics that an agent is dealing with. So if I'm looking at the... Most of the topics that are coming through, asset registers, approved maintenance, maintenance kits, service kit, kind of work orders, dashboards. These are all things that we've decided are in scope topics. But we have a new, set of topics with 33 of which are out of scope. So here, we can click here. So this is looking at an example conversation that that it... It's having where the topic is operation after unresolved equipment warning. So, here we can see this key exchange. It identified that this is probably not what it should be doing. So, it looks like the user's asking if I should... If it should... If, we should shut down a machine in the plant, and the agent is saying, yeah, leave it running. So that doesn't seem great. I'm gonna talk with Cobuild and try to, dive in a little deeper and try to figure out what's going on with these off topic requests. So Cobuild pops up and, it's... Cobuild is an agent itself that understands how to use and pull information from agent management. So Cobelt's like, yeah. Agent seems to have crossed from explaining equipment to authorizing operations, or not to authorizing operations, which is not good. And CodeBuild recommends, blocking the, blocking... In the future, changing this agent so it doesn't, try to tell you to take... Put machines up or take them down. It just provides advice on how to maintain them. So we can open a ServiceNow ticket, actually. So this is firing off a ticket to the remote team, and now the team add operating clearance guardrails to the maintenance, and it includes the selected exchange and the proposed guardrails and regression tests. So we have all of this, not just observability, but actionability built into the tool, which I I really think is quite important, to the whole thing. We'll be adding lots of additional features, you know, as we move on and evolve, inside of agent management. Some of the things that we're looking at, most recently or or most, specifically are around not just being able to, say, open tickets to have somebody deal with things in other systems, but going beyond just opening tickets to potentially having, like, a kill switch. If something seems like it's going really wrong, you could say auto shut off this agent. Right? Yeah. I burned $5,000,000 in a weekend, and it was gonna cost me $10,000. I probably just wanna have that thing to get turned off automatically, things like that. In addition, potentially being able to, have agent management be smart enough to make updates to third party agents. You can obviously make updates to data at Google agents, but third party ones as well is is is something we're looking at. One important point that I wanted to emphasize, and then I'm gonna, don't worry. I'm gonna look at all your questions, which I'm assuming there are several in the chat here. But a point that I wanted to emphasize is that Dataiku agent management is not an AI gateway. That's a really important thing to understand. An AI gateway would require that the traffic from all of these different platforms run through its little funnel. And then the way the gateway would understand what's going on is it would basically be reading all of that network traffic. Right? We are not requiring that. We don't think... We we think that in many cases, an AI gateway is a good idea, but if you're trying to have a global, manager like this, an AI gateway is too restricting. You're not gonna ever gonna get everyone to agree to route all their traffic through a single location. For one thing, it's a single point of failure. What we're doing instead is we are going out. We're reaching out to all of these third party platforms, identifying the way they store their telemetry, and believe me, they all store to their telemetry in a different way. It's incredibly annoying. And then pulling that telemetry data into a centralized location, storing it, trying to grab as much of the same data as possible across these different platforms so I can have kind of these global views, and then turning that into actionable insights. So... Yeah. I hope you enjoyed this. I can dive in deeper on anything else you wanna see. This is, you know, this is real stuff. These are real agents. It's not fake. So if you wanted me to click on other parts of the map, it doesn't break. So I'm gonna swap back up and pause my screen sharing and take a look at some questions here. Yeah. And while you do that, I'll quickly go back, just to share everyone. Again, we're really excited about this. Hopefully, some or many of you on the on the webinar were either with us in New York or in Paris, where we... At our succeed event where we talked a lot about agent management and, had a lot of good discussions. But, again, we're really excited about this. We think it's important to understand your entire agent landscape across, all the different frameworks that exist, understand your agent performance, your value, your accountability, and, really get a handle on that on that risk profile. So, we're very, very close to full release. You see a GA date there, but, we're we're already having a lot of conversations with customers, and, hopefully, we'll have some with many of you on this webinar. So, yeah, let's, let's open up for questions. You can either type them in the q and a, or the messages, and Jed and I will kinda rattle them off. I don't know, if you pulled any Jed from the existing chat already. There's a bunch here. I guess, a very basic one. Will a recording of this webinar be shared? Yes. That's an easy one. I'll take that one. I think this is an interesting one. What agents are running and who is accountable? So we just did our customer advisory board, yesterday with a bunch of our senior customers, and this question of accountability came up a lot. It... It's it's one I would almost turn back to the group because it's so interesting. Basically, let's let's take that shop floor agent. Right? IT, I guess, is accountable for keeping the agent up and running. So maybe if I... I'm gonna share my screen again really quick. I'm sorry. Yeah. I'll stop sharing. Because there... There's a bit of a dual accountability here. Where am I? I am. Okay. There we go. So if I'm looking at... Let's look at this this this one again. I'm looking at this. IT is really responsible for the operational side of the accountability. If it's up, kind of how much it's costing potentially, token usage, responses, all of that. Right? Is it running? But the business is really responsible for both identifying the KPIs that should be tracked, but then also ensuring that the agent is continuing to do what it wants or what what they want it to do. So it's... There's a dual ownership here between IT and business. IT is gonna be providing the platforms themselves where business is really going to be the ones building the agents and, defining what success looks like for those agents. So cool question. I like that. Yeah. And I I I love that, Jen. And, again, I think it's sort of core to Dataiku has always been bringing folks together. But I think with agents, it's it's even even more complicated as you said. Like, you've got IT that's responsible for the uptime. You've got the line of business that owns the the p and l and the ROI, and then you've got risk teams that, you know, are trying to do their jobs and mitigate risk. And I think it, agent management is a great way to unify all those teams in in one single view. So we have a question here. Do you define alert rules, or are they predefined? So there's layers to this. Some alert rules are predefined, meaning you can say, alright. Globally, when a, when cost jumps by more than 20% in a week, no matter what agent that is or what platform or infrastructure it's on, I want that to be a rule no matter what. I want an alert there. You can also define alerts at the infrastructure level level and then it'll... It would apply to all of the agents on that infrastructure. And then you can define very specific, alerts. So get really down into both the telemetry data and other associated data that you have for an agent, and build alerts completely bespoke around around the... Those agents. The fourth way the... The fourth thing you can do is you can actually use, some some of you might know about Dataiku Headless, but you can use Dataiku Headless to automatically generate bespoke business insights for your agents, underneath the covers. So that's basically say, hey, Headless, look at the telemetry data. What what business insights could I... Could and should I be tracking, and then generate the underlying necessary analytics flows to be able to track those agents, and that'll do it, which is cool. So, for example, this business alert, the one that we were just looking at, This unsafe operating advice, this was recommended by, Dataiku Headless. Basically, it was looking at off topic... Effectively, this is tracking when... Like, how many off topic conversations are happening and then identifying what type of off topic conversation it was, which is unsafe operating device. Let's see. There's another question here. What is the added cost to run Dataiku agents? I'm not entirely sure of of the concept of that question, but I I do think it's an important one. So, to be clear, reaffirming our agnosticism, an agent as an agent. There's no, you know, data cu agents are free because we really want you to build in cu agents. Although, I don't know what you said. But, yeah, it... It's sort of agnostic to the agent. Doesn't matter. You know? Certainly, we're most familiar with Dataiku agents, but, we are very much focused on any agent from any of our supported frameworks to be able to give you that holistic picture there. I see one here. How do you identify what was the response time using agents developed against the prompts used to use that agent. I I don't know if I'm with the prompt. Developed against the prompts used to use. Like, I I... Maybe it's a question of, like, is response time really a useful metric when different prompts can deliver widely... Wildly different response times. Yeah. That's potentially true. Of course, it's also true that not all agents that we're tracking here are conversational necessarily. You could have completely autonomous, always on running agents that are not responding to human beings' requests, but responding to, I don't know, a document being loaded somewhere. Right? Or a insurance claim coming through, whatever that might be. So in that case, the initial input is going to be, kind of more repeatable than a human being making a request. So... There's a, Yeah. Topics. question there and then... The topics listed with each agent. automatically generated. Yes. So we have a topic generator or it's a topic generation recipe underneath underneath the covers, that auto... Basically, what... What's happening, pretty cool. Once you have a good bit of data, let's say, a couple 100, agent, telemetry logs, for for different agent runs. We scan the... That information and then identify the main topics that are coming through. So you can see the top seven or eight main topics that are coming through. At that point, a user would have to define whether or not those topics were in scope or not. The way you define where the topics are in scope is you can either manually label all those topics as being in scope, or you can give a description of what the agent should be doing. So for example, this agent that we've been looking at, it should be providing maintenance advice. It should not be telling people to leave the machine on or take it down. Right? It should always be leaving that decision to a human being. So you can define the scope or the intended use of an agent, and then as new topics come in and get identified, we're constantly repolling for new topics that are coming through or adding conversations to existing topics. New topics come through, that gets compared to the scope and then defined as in in scope or out of scope. Hope that answers the question. Follow-up on accountability. I think your product is... Service is not so much for the tech side, but for the CTOs, CIOs because this business will be liable, and for them, this is prominent, probably the most valuable tool to define it than at risk. Yes. I agree. My question. What is your thought in expanding this to a type of AI gateway or AI firewall, if you will? Because if the info escapes, it is too late. We are looking down the line. I don't wanna put exact times on it, but, down the line at also providing, if if people want, an AI gateway. That is that is something that we're looking at. Definitely. Yeah. But I also think, you know, what you shared earlier, Jed, about, like, this isn't, that, but I do think, you know, many folks, in the conversation I've had, like, see the value in what this is bringing to bear, where it's not necessarily. true live, but more so like a portfolio manager, and and an overall, assistant. Yeah. There's a question, what risk categories are you tracking? So we come with a default. This is the default risk taxonomy right here. So across operational reliability, autonomy and scope control, security and adversarial robustness, compliance and legal exposure, and then harm its output safety. But, all of these, as you can see, you can add additional risks or add categories that match your company's needs or expectations. Let's see. How do you validate security? Example validating whether user can access the data or share the date or user is uploading any documents which contains PII So the security stuff is really quite. difficult. We can identify misuse. It's more difficult to identify, like, user access permissions. Right? So we know what users are doing, but unless the third party platform, also provides information around the security perimeters that users... Like, what what what type of data users have... Should have access to. It's a... That's a that's a very tough one for us to be able to do right now. It's something we're looking into. One thing you can do, I think, let's do it and see if I have a good example here. Okay. This is a cool one. If we look at the agent structure here, this is like a pretty complex agent, so a lot of different things going on. But the structure of this agent... Here, it's hitting a specific knowledge bank. So we do know what, like, what knowledge banks or databases an agent has access to. It's more we don't know what the, limits of the users of that agent, what what they should be getting access to. We're looking at how to do that. Very challenging. Everybody kinda stores that information in a wide, disparate list of places. But, yeah, still an unsolved. problem. And and, also, I I do think, you know, some of those questions, likely come down to, like, where the agents were developed. And, Yes. you know, like, within Dataiku, we have a lot of those capabilities to, you know, ensure that, data sharings, to your expectations, PII detection with prompts, and things like that. And so I think a lot of this comes down to the agent framework, and then agent management is sort of a catch and monitor to ensure that, you know, there's no leaks. Absolutely. Any others that you're seeing? Is your product, so I could talk a little bit about the architecture, because that's... We have a question around, is is it standalone software, or do I need to have, like, additional MCP servers or things like that? Do it as much as possible. We we we do not want to force our... The users of, the agent management to, for example, need to inject hooks into all of their third party agents because we don't think people will do it. DataEye, we want this to be very much a stand alone stand alone piece of software that proactively investigates all these third party platforms and identifies the agents without the people who are building the agents on those platforms needing to do anything. Underneath the covers, data, agent management is connected to a AI service. So here, you can run the AI services in two different modes. You can either run it in bring your own LLM. So here I am bringing my own LLM. Or you can run it in Dataiku... Operated by Dataiku, in which case we will manage, which underlying models and, actions are being done. Both work great. But, yeah. For lots of things inside agent management, we are, as I've mentioned before, we are using agents ourselves to identify topics, to give you co build, to... For all of those different capabilities. And, yeah, that's kind of part and parcel of of what's going on. Oh, that's a good one. What mistakes do you see clients typically see when using agents? Like, let's see. What... What's been interesting for me? I I think the, I don't know if it's mistakes, but I really love the auto generated topics, partially because, like, sometimes it it identifies things that are bad and that you need to fix, like one of the example I showed you. But it also often shows me kind of ingenious topics that I'm like, oh, wow. The agent should be able to do that. So it helps, encourage new... Kind of new feature. In many ways, off topic requests are feature requests is how I think of it. Right? And our off topic conversation is in either a feature request or a... Something that I need to fix. But the feature request part is pretty interesting. I personally... Probably the most common thing is cost. Yep. Realizing that, like, even, like, looking like like this type of cost, so, like, cost by platform, realizing that Snowflake is five times as expensive as everything else is, in and of itself, interesting when they're... When you're laid up across all of these things. So, like, your... That... This initial immediate impact, when we turn it on for customers and they see the cost of everything across the different infrastructures, that's a big light bulb moment for them. And often, there's, like, an immediate phone call or email, to to to resolve something or, you know, just that kind of transparency is really great. One thing I'll say, which is maybe slightly related to to agent management, but, you know, back to my earlier example, you know, some organizations say, like, hey. We we have an AI account. So I'm like, you will not say the word agent until you've, you know, come to us and told us what you're going to do and all of those things. I I often find those as as mistakes because that's a good way to turn people away and get shadow AI in your organization. And so I think a lot of folks are are saying, like, hey. And and then at the same point, like, assuming that the only agents exist are the ones that have gone through this rigorous counsel because, you know, the beauty of this inventorying process and the scanning, is that, you know, people likely will not go to that counsel until they're ready. Right? They don't wanna waste your time. They don't wanna do all the work until, they've actually, you know, have something worthwhile to say. So, like, being able to, enable this innovation and just know that, like, you're gonna get eyes on it. If they are working in this framework, you'll see it on your radar. May not be ready. It may not be production, quote, unquote, but at least you'll have visibility to track it and make sure that it does go through your counsel and that. it does have all the appropriate approvals. Because, again, it's just people find a way. That... That's, you know, humanity. It's like Jurassic a way. Park. Jurassic finds a way. Yeah. Absolutely. Park. Well, cool. I really appreciate all these great questions. The lively chat is always, always nice. Any, final thoughts, Austin? I don't know. You guys are onto something big. I think I think Thanks. that says it all. Again, appreciate everyone joining. Appreciate the feedback. Stay tuned for the recording, the slides, which will be shared for you. And, yes, please, please reach out, and happy to chat with more on agent management. Thank you all.