Video: Uptime is a lie: why traditional observability fails for AI agents | Duration: 3292s | Summary: Uptime is a lie: why traditional observability fails for AI agents | Chapters: Welcome and Introductions (2.88s), AI Infrastructure Governance (72.235s), AI Observability Focus (195.22s), Agentic AI Risks (316.425s), Agent Observability Challenges (511.095s), Agent Types Evolution (780.08s), Simple Agent Architecture (912.9s), Structured Agent Workflows (1052.74s), Monitoring Agentic Systems (1337.15s), Agent Behavioral Tracking (1863.775s), Self-Managing Agents (2269.01s), Final Thoughts (2459.92s), Q&A and Closing (2818.015s)
Transcript for "Uptime is a lie: why traditional observability fails for AI agents": Alright. Alright. Let's kick this off. Welcome. Welcome to this session. Good morning, good evening, depending on where you are. Thanks for joining us. Today is gonna be important. I think this is a very nice topic. I think this is the time for us to talk about this because it's happening, and we are seeing it in the industry and in these deployments. So today, I am super happy to say that we have a co speaker. I'm gonna be guiding the conversation. I'm Katharina Herrera. I'm field chief data officer at Heroku. And what that means is that I spend a lot of my weeks inside Fortune 500 AI programs, the ones that are going very well and also the ones that are not going that well. Our goal today is to share of those insights. I'm super happy to say, Jed, welcome. Did you want to say hello and introduce yourself very quickly? Sure. I'm Jed. I'm a SVP van platform at Dataiku. I help build the tool and make sure our customers are getting the maximum amount of value out of it. Exactly. Mhmm. Together, we're going to be talking about a very important topic and trying to close a little bit the gap between what happens from the infrastructure monitoring perspective versus the decision evaluation perspective. And for that, I will quickly have a little bit of a recap for the last twenty, thirty years. We have all the hard work going on, all of this compute happening and exponentially getting better and better. But at the same time, us trying to ask questions to this data and trying to make the most out of it with any AI technique. So at the end, what works in an enterprise deployment at production at a scale is when you have that domain expert prompting, asking the question that is coming from, from that expertise. When you have every piece of the artifact from the data silo to everything you needed to do to transform it into an AI artifact. So we can call that Descriptive Analytics, Predictive Analytics, Generative AI, Agentic AI. Everything needs to be orchestrated within a framework and that's part of what we're gonna highlight today. That is not a black box. It's a governance framework that is going to allow us to collaborate. When I say to collaborate, is that recently, the last two years, we have different teams asking different questions regarding these AI outcomes. Why? Because not the fact that it's a green dashboard or a check mark means that the agent is actually providing an accurate response, and that's part of what we're going to highlight today during this webinar. But when you combine all of these variables considering every single component of it, we can think about a deployment that is going to be in a production enterprise environment at a scale, where we are going to be able to collaborate towards that business use case goal. Ultimately, let's recap. There is a lot that has been going on in the last twenty, thirty years. We have the data layer that you have been able to control from a long time. Now you have the compute layer that now you are able to even tune even better with the commodity GPUs that we have available in the market. Now you have the intelligence layer where you have all of these generative AI ecosystem and all of these LLMs models. Ultimately, you have all of these teams asking questions to these data under the AI umbrella and descriptive, predictive, generative, agentic workflows that need to be governed. So very important to think about the right side of the circle because usually from the tech aspect, we focus too much on the left side. And right now, I will say today's observability topic is actually highlighting what happens when you deploy these, how do you keep an eye on these, how do you know if this is still accurate, how do you know if you can actually rely on that answer that this agent is generating that can be potentially customer facing reply. So today, we're gonna focus on that observability level. And a little bit of the context is that this is changing and this is happening. I mean, two, three years ago, we were introduced to this Gen AI ecosystem. Now we have all of these LLMs available in the market. And now we have to think about what needs to happen for us to trust these results, to ensure our guardrails are in place, and perhaps to rethink about the guardrails that we are adding because traditional methods may not be for this new layer of generative AI and agentic AI workflows. That's what we are covering today. To kick it off, I'm just going to say that the last year has been very exciting. We have a lot going on in the field and a lot of agentic experiments going on on top of a very silo infrastructure. We can think about as a piece of a genetic flow on top of that silo tech that is asking questions to on top of those descriptive predictive generative assets that are targeting these business use cases. We know it's getting, demanded by the field. We know teams are actually looking forward to add this level of technology into this workflows because it accelerates the decision making process tremendously. It empowers these SMEs tremendously. But at the same time, you have some risks going on as well as you get into bigger and bigger deployments. And we are talking about hundreds of agents, thousands of agents that, yeah, maybe answering all of these questions are coming from your teams and from your domain experts. But how do you know that the answer is actually accurate, that you can trust it? So today, we're gonna focus on the risk aspect of this, and we're gonna rethink about what traditionally has been going on in terms of how do we ensure that we have some kind of test where we say, okay, that is accurate, is working, not working. And part of what we're gonna talk about today is that perhaps the way that we have been using these frameworks to check working, not working is not longer practical when you have nondeterministic aspects as part of this workflow. And when we talk about nondeterministic aspects, we know a genetic AI relies on a generative AI ecosystem. And that generative AI ecosystem relies on these language models that at the end are generating an answer that may or may not be accurate. You have all of these dashboards that we have been collecting and they may show all greens. This is working. This is working. This is working. Yes. Yes. Yes. That doesn't mean it's right. An example, think about your favorite restaurant. Everybody there? I'm thinking about my favorite restaurant. Now think about the kitchen in that restaurant and how the dashboard looks like. It's like, okay. It's working. We have the chef going. He's moving super fast. Everybody in the team is cutting and preparing. Everybody's super fast. All the barns are on in the kitchen. The ticket printer is giving me the orders. Everything is green, green, green, dashboard, dashboard, dashboard. The truth is when you open the door, all the customers are super frustrated. You were sending raw chicken. You changed the ice cream for the meat sauce, and it's a disaster. So not because you have a green flag in a dashboard, not because you have an okay on a test means that you are actually measuring what it needs to be measured. No? So the 100% uptime, the low latency, the serial server errors, all of that is gonna be subject to what the workflow is doing, what the agent agentic use case is targeting, and how you're going to define what observability means for you, and how you're going to define whether the machine is running, yes, but it also is this running correctly, is the agent actually doing its job. And that's what we're gonna deep dive today. So I cover overall where we are, what has been going on in the industry, and why the demand for this observability framework has been increasing tremendously. I will say that we've seen more and more agents deployed, a lot of experiments going on in silos. We see a lot of these platforms with agentic behavior on top or new agentic features on top. But at the end, you need to ensure that you are thinking about the production deployment, how this is going to maintain reliable, how do we trust these results? Is this transparent? Is this something that I can put my name behind during a board meeting? Do I feel that I know what is going on behind the scenes? And if we think about everything that is going on in the field, we can group this into three different categories that we wanted to highlight today as heads up. Not because you have a check mark on uptime or latency or error rate, you are defining if the agent is giving you the correct answers. Drift, for example, very generally speaking, a GPS system, you think that is giving you the navigation, you go and if this is a new city that you are navigating, but the map has not been updated in months. So you have wrong information. The agent is giving you an answer, but you are driving straight into a brand new one way street. You have that drift behind the scenes. Or perhaps you have an older enterprise policy that is part of the documents that the agent is leveraging as part of the knowledge bank. And still, it's gonna give you an answer. It doesn't mean that it's right. So when you have a scope creep example, when you have a customer chatting and saying, hey. My order was late. Can you give me a discount or a refund? Well, you're going to have a test that is giving you, you have every API call succeed, you have a 200 okay HTTP, you're checking everything that you need to check from the tool perspective, but then you say, hey, you were not allowed to do refunds, agent. Now I have a Humana's bill and everything is because you approved the unapproved refunds just because I didn't define those guardrails properly, and you were taking over responsibilities that were not part of your role. Another error that can be very significant in terms of the outcomes. Decision quality failure, for example. Let's say that you are going to have a retrieval augmented generation bot that is using the multi nodal knowledge bank when you have a structure and a structure information as part of the assets. And let's say that the bot scanned the document, you have a PDF, and then you have a PDF that had a 100 number. And you had an indemnification that was capped at $10,000,000, but when you did the conversion of the on a structured data point, you have a $10,000 because it was a mistake on that zero. Something like that. Right? So you're gonna have an agent that use struggling with the formatting of the artifact and now is completely misjudging the liability boundary. See? So it completely depends on the use case. It completely depends on all those guardrails that you have to define thinking about that use case, that you have to define thinking about what is happening today as part of this workflows and not so much into what has been going on traditionally. No? So I think this is a time where we're gonna hear a little more, Jen, I would say. This is an important moment where we acknowledge the shift into what needs to happen for these newer workflows that contain generative AI as part of these assets. So Renee, if you want to highlight a little bit the difference between these two and perhaps show a couple of examples of what we're thinking about here. Sure. Thanks, Catalina, for that very nice introduction. We've seen a few evolutions in the well, even in the definition of what an agent is over the last couple years. I think that two years ago, maybe even a year ago, 90%, 95% of the agents were effectively conversational agents. And what I mean by conversational agent is, a user asks the agent for advice or support or to perform some kind of action, And then the agent chooses from a set of tools available to it, performs that action, and then returns the result to the user. And this is a pretty simple structure, I think. It's a pretty straightforward structure in that, there's this the user is built into the loop in every case. Let me, let me pull something up here really quick. Walk through this in a little more detail. So we've got two different types of agents here. I'm a try to share my screen. Somehow. Probably. Great. Alright. So we should see there we go. Got a nice this Dataiku. We won't dive into Dataiku too much, but but I'm gonna talk about the difference between a simple agent and what we're calling structured agents, which we can talk in a little more detail about in just a minute. So a simple agent here let's just look at the view of it. The simple agent is what I was speaking of before. Essentially, a user will ask a question here to this loan officer agent and the loan officer agent will interpret that question and say, okay. Either it asked me to predict whether somebody's gonna default on their loan or it asked me to create a ticket or it asked me to look up information here in Snowflake with this loan information expert. And then it's gonna get the information back and send it back to the user. And then the user is going to make some final action on top of it. Now an agent like this is relatively easy to make. It's relatively compact and easy to understand. And in many ways, it's sort of a reflection of maybe like an individual person inside your organization. One downside of this is that it's not incredibly easy to measure the ROI of an agent like this. At best, it basically augments the user who otherwise would have probably looked this information up on their own. And so maybe we're actually still tracking, like, the ROI of a human being. Maybe they move faster or something like that. But it's not augmenting or replacing business processes itself. A tracking and understanding whether this agent is successful or not is actually maybe not that difficult. You probably just ask the user who leverages it or you just get an NPS score for the set of users leveraging it. They say they like it, it's good. They say they don't, it's not. This is how people were building agents a year ago. Now what's changed in that last year? One of the big things that changed, is well, OpenClub actually kind of showed us the way for more autonomous workflows. And a more autonomous workflow really expands both the potential ROI, but also the potential risk of an agent. So let's swap over to another one here, and let's go look at a structured visual agent. So a structured visual agent is something that can run automatically without user input, and we're actually chaining together many underlying agents to represent a business workflow or a task workflow. So here, for example, every time a new loan application is a loan concierge agent. Basically, it's supposed to decide whether or not to approve a user from a loan and then go out and let the user know or escalate it. So we can look through we can look through how we structured this agent even though what it really represents is probably the augmented jobs of a set of different people inside of an organization, potentially people in different departments. So we can see look look at how it works here. A new dataset or, let's say, a new loan comes in here at the start. We intake that loan and look up the customer who's requesting that loan. So we get some basic information about that customer. Then we take that information from the customer and pass it into a parallel research block. This parallel research block can do multiple different things and you can see it's using different LLMs. So we'll use, like, a cheaper LLM for an easier task. So one is it does market research reaching out to Snowflake, so it's a third party tool. Another one is hitting a prediction model. So it's hitting, trying to get a prediction on whether they'll default just like what we did in the non structured one. And then a third checks policies, inside that might pertain to the user in question. So probably looking up restrictions on loan size by state or things like that. So all are pretty broad, individual agents. Like, you could see each of these as an agent itself. But we can combine them together in a term deterministic way that said, okay. These only get look used after we look up the customer information. We parallelize that information. We save the findings, and then we have a second reflection agent that reflects on those findings and tries to make a good decision on whether we should pass or fail the underlying user with the request for a loan. So it's reflected. Pass or fail, the results get compressed. The decision is extracted, and then we route the output of that decision to two different paths. One, where we generate an offer and send the offer letter to the end user, and another where we instead create a ServiceNow ticket that gets escalated to a, banker. So with this type of workflow, yes, we can call it an agent, but I really consider it more of an agentic system or an agentic task representation. We're mixing and matching deterministic workflows with agentic capabilities that can, on the fly, for example, generate a nice cover letter or do research of this type. This is really where we get bigger ROI gains. It's not in that just providing advice to an individual person for an individual problem. It's chaining together, the fully autonomous workflows to actually solve a problem. But, of course, as Catalina mentioned in her intro, this is also a lot more dangerous. Right? Here were this agent is deciding to give people money. That directly affects your bottom line. Right? So making sure that you have really, really good management and understanding of not just what the overall agent is doing but how well each one of these individual underlying components are working and potentially how this agent is changing which components it uses more over time is very, very critical. Imagine we have this agent in production and it was approving offers, at the end on, like, 60% of users and then all of a sudden it starts approving offers on 30 even though we didn't change anything, that would be a pretty big deal. Right? Like, we would want to be able to understand exactly why and how that changed. So as we move from con individual conversational agents to more structured task reproducing agents, the criticality of maintaining agent management, escalates drastically. These things are going to autonomously affect the bottom line of your business, hopefully very positively, but you need to be able to track and understand how they're doing so. Thanks. Thanks, Donna. I think what you are showing is so important right now because it highlights how much you can control that deterministic aspect of that workflow even if you have non deterministic components. And I think it's one of the number one issues that we see when we are trying to deploy this into production use cases. At the end, the agent has been trained depending on the LN that you are using. You may be using different trained data, different parameters, different answers, perhaps, even. You you may ask 10 questions to the same agent and get 10 different answers, and we see that. There are use cases where that cannot happen, period. Especially, if we think about the different verticals that are part of this. So one thing is to accelerate AI deployments in supply chain and move faster, but another thing is to be fully in compliance for the audit, that we have in the next month in pharma, for example. And that's where you start combining all of these variables, where all of these details matter and matter tremendously. Everybody can use the technology. Yes. It's there. Yes. We have generative AI ecosystems supporting all of these questions. But what we are showing right here right now is how the SME is the one that actually is being empowered by. So whoever knows about every single detail of the workflow and that process is that business user expert that is able to define these deterministic aspects on the non deterministic flow. Leverage agentic AI, but know exactly the guardrails that needs to exist for this to be accurate. Know who else is gonna be asking which other questions for this to be ready for a deployment production at a scale, and what that means for you and for your teams. Because you may be thinking about a flow from the technical aspect only, but the more generative AI you add to these agentic workflows, the more nondeterministic components, the more questions you will have to ask to somebody in the near future. Legal, hallucination, policies, compliances, audits. Are we aware of what we are deploying here? Right? So I think that your UX right here right there, it it summarizes, a lot of what this conversation is about. And I think you are trying two different features that I want to ensure that we highlight while we have this screen on. On the right side of this screen, Jare is actually working with natural language, defining the business use case using English, one paragraph that comes from your domain expert. That domain expert is typing that in natural language using Co Build, which is our near future as part of Dataiku. Dataiku is going to grab that prompt and it's going to give you visually every single node as part of the workflow for that prompt to produce that end result. So in the middle of the screen, Jered is showing the little boxes that are pretty much the stages that a agentic workflow is following depending on the guardrails that need to be part of this and depending on the business domain, expert owners, security permissions, everything that is embedded and included. But at the same time, it's allowing somebody that is not 100% technical to make use of the technology, to leverage the genetic ecosystem, to leverage the generative ecosystem, but in a way where you can actually sign your name and say, no, this is not a black box. I know what is going on behind the scenes. I know that I am passing my own IS test, explainability test, transparency test, and everything else that I need to follow internally for this to be checked for go into production environment. Right, Jess. I don't know if you want to share anything else there. We want to show the next slide that is a little graphical into different metrics that we can collect out of this. Now let's, go to the next slide. You can see I'm adding some test cases right now to this, SVA. As I mentioned, it's quite important to be able to test to make sure it's all working. So I can just ask it to do that. But, yeah, let's go to the next slide. That's beautiful. So if you stop sharing for me a second, let me take it over. Perfect. I think that that was very visual and it is part of how we wanted to finish this slide. The goal here is to highlight that we have two paths and the things are changing. We cannot think purely deterministic because we are not working with purely deterministic artifacts, period. So the more that, we are aware of that and the more we define those guardrails and those deterministic blocks, the better you're going to be able to connect this with the actual use case, with the actual ROI. Ultimately, you're going to be facing different failure modes and you're going to be facing different categories that you have to be on top of when you are deploying thousands of these agents into production. We are talking about different things. One thing, yes, Jie, do you want to share something? I hear you. No, you're good. We're talking about two things. One thing is, hey, I understand what is going on behind the scenes and I have a dashboard. Yes, acceleration, speed, latency, whatever else I was checking three years ago. But now I am aware that I am using non deterministic components. Therefore, a new set of metrics, should be available for us to keep an eye on this. Right? So what else can we measure here, and what else should we be thinking about here, And how we in Dataiku are thinking about this? So I wanted to introduce this slide as a alternative use of, metrics that actually are gonna give us a more accurate workflow probability within the non deterministic components that generative AI includes into these ecosystems. No? So when you are thinking about this decision system, as Jen mentioned it, you have those blocks, but at the same time you are focusing on something else besides the infrastructure metrics that you have been collecting before. I'm going to stop right here. I'm sure Jeff, do you want to highlight anything here? Did you want to show anything in terms of the UX here? Why don't you walk through it and then I'll pull up, the UX in a minute? Perfect. What this slide is telling us is that within the deterministic flows, we have been creating dashboards and metrics that are very tied to latency uptime, the HTTP status codes, the 200 versus the 500, the talking outputs, the CPUs, memory usage. All of this was completely valid when we had all of these AI machine learning workflows that were reusing assets that were mostly descriptive and predictive. So you are focusing very on the machine itself and the latency and the times. Now, when you have a component that is not purely deterministic and you are using this generative AI ecosystem, there are a whole new set of ways that we can actually kind of align into, is this working? Yes, no, but is it working correctly? And that is part of what we are bringing right now into the table. Right? So how can we leverage this central repository where we have all of these agents, popping from the different tech pockets across the enterprise. Every agent is bringing a whole set of telemetry, and we we know what is going on here. How are we gonna kind of align on what needs to happen for us in terms of metrics to say that this is working? So I wanted to share this new set of categories. And, if, you want me to stop for a second and see how are we thinking about this and show a little bit of these metrics coming from this telemetry, that will be useful. Alright. Sure Stop sharing for me a second. Yep. You got it. Yeah. Great. So as we're trying to track agents across many different platforms, kinda like to add a nice point of pointing out, us and every other vendor has sort of created a challenging situation for for, for organizations right now. And that we all are telling our clients that you should be building agents inside of our platform. We all say that. Yeah. Do it. It's great. Not just us, but the three major clouds, Snowflake, Databricks, ServiceNow, Salesforce, you name it. We're all gonna be offering the ability to build agents inside of our platforms. So as these agents are being built all over the place and as, well, you just saw they're becoming more autonomous and less just chatbots that assist you, you do really need to understand not just if they're working, but if they're doing what you expect. I think that's kind of the key here. Let's take a look at what that might mean for a specific agent. First off, so we could see we've connected to I think we have seven different platforms here. If we want, we could connect to another platform. Sure. It's working. That's good. So I'm pulling in some additional agents from Databricks. I don't care about you. Import these. Now I have my inventory of all my different agents. So let's look at what, like, the initial set of information we might look at from a given agent. Take a look at this KYC verification agent right here. So we can see here this operational health. So this is basically like uptime. This is sort of the equivalent. Like, if an agent was just an API, what would I be checking? I can get that info right here. I can also get things like usage statistics. So how many people are hitting it, daily, weekly, monthly. Okay. Great to know. And then I can get things like token usage, so I understand approximately how much it's costing me. But the interesting one to me is this behavioral tracking. All of these are the same type of things that you might track with a regular API. But behavioral stability or behavioral tracking, KPI tracking, that's a new thing for agents. Agent behavior can change over time for a variety of reasons, either model shift or data shift or, user action shift. All can affect the way that a model or an agent is behaving in a way that an API is not ever gonna be affected by. So while this KYC verification appears to be up and operational, 99.98% operational uptime, that's pretty good, I can go ahead and review this behavioral stability in more detail to understand what's actually happening over time. So we see that the outcome remember, this is sort of like similar to that loan workflow that I was walking through before. Here we could see this is a know your customer. So we could see a manually manual escalation rate has climbed from 8% to 23% since September. So the agent agent is routing more, credit applications to manually review and that's slowing down processing. Good time. It's costing us money. So what is happening there? Can we get a deeper view of how this agent is changing? I can drop in and investigate the behavior of my agent which basically tracks and again, think about that task view that I showed you before. It tracks which sub tools and paths this agent is taking. And as it starts taking new sub tools or different paths, we can see that as a behavioral change. So here we can see this agent, passes information to an identity verifier tool, and now specifically a non US passport sub agent is pushing way more records to manual escalation. US Passport Agents are the same, but a non passport agent is pushing a lot more to regulation, thereby costing us money. So I would like to do something about that, potentially request a review from the AIML team to take a look at this and try to revert it, or if necessary, keep it and change our expectations on how this agent's gonna work in the long term. But if I hadn't had this behavioral tracking in the system, all of this would have been invisible. I would have just been losing money or, making more work for people, basically having my agent be less effective than it could have been. This is critical especially as you move from, you know, 10 agents to potentially thousands of agents. Not all agents are, you know, created equal or are gonna be equally important, or are gonna cost the same amount. But for those that are critical and those that are very automated, you really, really want these more advanced metrics and checks in place. We don't necessarily believe that every single agent needs this. Most agents, maybe, I don't know, most, but many agents can just say, okay. This is this it's up and down and this is how much it's costing and your end user can decide whether it's valuable or not. But for the ones that are truly autonomous, the ones that are making business decisions and tenure or on your behalf, you really do need to be able to track their behavior over time. It's, similar to how you would always want to be able to track the behavior of your employees, how many employees you have, whether they're, performing to the best of their abilities. You wanna do the same thing with agents. Simply knowing whether someone showed up to work or logged into Slack is like, yeah. You wanna know that. But that's the bare minimum. Right? That's the same thing as kind of having agent uptime. You also want to give them KPIs, and you wanna be able to track how track how they're doing. And, ideally, you want some way of, like, even without having to manually set a KPI for everything, you wanna know if the way they're acting or the way they're performing is changing. And that's why that that, being able to track the underlying tool use ends up being really critical. One thing about a system like this, about a monitoring system, like like, what what I'm showing you right here, the agent management, is that there's a there's a challenge that I think folks who have ever dealt with, let's say, cataloging tools might be a little familiar with. And the the challenge here is that a tool like agent management is only useful so long as there's information in it. It's only useful so long as that the KPIs for each agent are registered. The owner of each agent is written down. And in some cases, the onus and the pressure of maintaining the meta information about every one of these agents costs more than the worth of knowing that information, provides to your business, or it's on too small a set of IT folks to try to maintain a platform like this. And then that ends up very much being a blocker for this this type of thing in reality. So while it can look very good on paper when I show you this demo right here, the reality is that 90% of this stuff never gets filled out and it sits there as a blank platform. We really don't want that to happen to you or for you. We wanna make the systems, work in such a way that the agent management system fills itself out in every way possible. So what what is good for automating, tasks like that? An agent. So a core part of any agent management system should be a very experienced agent that knows how to look at the logs of every single agent that comes through from across all these different platforms, see what's available there, see what you could be tracking from a KPI status, and then automatically build out all of those tracking features. This way look. If you let's say it takes an hour to manually set up a given agent so that it could do what I just showed you in inside of agent management. Well, if you have 2,500 agents, that's twenty five hundred hours of work to keep the thing going, plus there's new agents coming in all the time. So it's absolutely critical if you're not gonna have managing your agents be a massive full time job for a whole team of people, but instead something that happens relatively autonomously to have an agent that manages agents. I know agents all the way down, but it's actually something that an agent is quite good at. So that's also what we've been building into Dataiku agent management is we need an agent that assesses agent quality. And, yes, it does look at itself. So doctor Hill, thyself does happen. Back to you, Catalina, for, for any final thoughts here. Yeah. Yeah. I know. I I love that, and I think that's the perfect introduction to the next slide if you let me share. And it's the agent. looking into agents, and I think that goes into how they, like, who thinks about this holistically. And I would say this is start with saying that we have been thinking about this for years, that agentic AI is just the newest layer there, but this is this has been there for years. Right? So you cannot govern agents using infrastructure tools alone, period. You cannot govern them using a static AI frameworks because that doesn't work anymore. That is a world that doesn't exist anymore. For us, this is a multivariable problem. The first thing is that this is a multivariable, thinking about the personas that are involved in this workflow, the orchestration of all the artifacts and the governance framework that you need to have for these personas to collaborate. Number one. Number two, you are adding non deterministic artifacts as part of these workflows. And that means that there is a lot of responsibility here in terms of what we do as the humans empowered by this technology, but we are all thinking slightly differently because we all have different OKRs to oversee. So the number one assumption here is that you are no longer the only one who cares about that data outcome. And I come from my data science background. I am a data scientist. And I used to think that way until I realized that it's half of the equation. Right? So you think about who cares about this data outcome and how they care about this differently than me. Therefore, we can holistically achieve that ROI moving together towards this framework. What that means is that you will have an IT engineer that is thinking about the inventory, the health monitoring, the whatever topic drift, tool drift, and whatever else is here or seen. But if you have a business owner, you have a slightly different metrics. You require a tracking of decision quality, task accuracy, cost per session, measurable ROI, like where is this going? Right? But you may have a completely different persona here that cares about the hallucination that is responding to the customer facing question that matters. Right? So that compliance and legal team that is gonna be me demanding that audit, the drills, the risk classification, the approval, workflow, documentation, the who is the person that is the owner of this workflow or this initiative? Do we have a stakeholder If we need to retrain the agent, who needs to sign this off so we can redeploy it in production? Right? We need to start thinking bigger, bigger at a scale. So the cost management the agent management platform is gonna unify these personas. It's going to allow you to move beyond the basic systems metrics to continuously score the decision, validate, to actually inspect the reasoning chain, and to enforce these enterprise guardrails because at the end, we we want to think about this at a scale production at a scale. As Jeff was showing in the screen, we have thousands of agents that are being deployed around. The way we think about this is that we want to measure what the agent decides. We wanna measure that is right, not only from the yes, no tool perspective, but capturing the correctness and the logic behind the scenes that your knowledge that comes from your SMEs, that comes from your domain expertise for the how you are working on those deterministic subflows within the workflow that is leveraging generative AI. And they also wanna monitor these multiple agents at a scale. You need to think about who else is using these, how many else, like, how many else, how many questions I can answer with the same workflow, How many of these artifacts are actually reusable as part of this question? That's where we start moving that ROI, ROI, ROI. Solution number one, you are going to be on top of every change in ecosystem. Change is the only constant in terms of that infrastructure. Your goal is to ensure that you are doing the best in every layer that you can control, data compute intelligence. Number two is that you have to provide the framework that is gonna empower that SME to leverage the domain expertise within these workflows that, yes, they're gonna be micro automation jobs at some point, but you are gonna have to control very nicely all the guardrails that you are adding on top of these non deterministic aspects. And that's where we can walk into a board meeting and we can say, yes, we know what's going on behind the scenes. We know exactly what agents are running. We know how they are delivering what they are delivering. We know what is the performance. We own a clear roadmap of risk and opportunity. Different OKRs because, holistically, we are a framework of diverse personas working together and leveraging this AI. And at the end, you want this to be as clear as possible for you and for your teams to move with confidence. No? So that's how Dataiku thinks about agents. That's how Dataiku is actually leveraging all of these different layers of the ecosystem, especially the last one, which is the agentic flows with these LLMs as part of this ecosystem. I'm going to ensure that we have the opportunity to answer any questions from the audience being very aware of the time. I'm going to open the floor for questions at this moment. This is your opportunity. Any questions? I think James has a nice question here that's probably more general, which is, if if they wanna know more, how should they connect to us to have a deeper conversation? What's the right way to go? That is a great question. I'm going to start with this nice screenshot. James, it would be awesome to have you as part of this crew. So September 24 in New York in Spring Studios, there is going to be about a thousand of your peers across different verticals. The goal of these scenarios is to actually share and learn together. Right? We are going through a very important chapter in humanity, I will say, and this is a moment where we know what's working, what's not. How can I think about this differently? How can I empower my SMEs in a framework that at the same time is giving me the guardrails that I feel confident with? So Screenshot this, take a picture of the QR. This is a great opportunity to meet with peers and to share what's working, what's not, how can I think about this differently? Now in addition to that, I will say that we should have follow-up e mail sent and that you can actually reply to that as well if we need an additional follow-up. Let's see what is the next question here. Look at that. I meant that every chance. I can speak to this. So the question, in the chat here is should organize organizations look to augment every job function in this way? Where's the risk of technical debt in this type of tech work? I think that's a really great question. I think that there is definitely a hierarchy of lower hanging versus more difficult fruit to harvest. A lot of where I've seen autonomous agents be most successful right now, especially as people are beginning to roll things out is really in maintenance work. So for example, in Dataiku, we have a autonomous agent that looks at every single we have hundreds of machine learning, models running live on API services that do predict everything from, you know, customer revenue to churn to, AE success to targeted, outcome of when we are releasing our next feature, etcetera. So these are all running on these big Kubernetes clusters, and we have an automated agent that is constantly scanning those clusters and identifying when, a model is going when a model is down and then attempting to figure out what's wrong with the cluster or the model or whatever to fix it. And then if it can't fix it, then it escalates to a human. But like these type of maintenance tasks, basically things that are difficult in a somewhat technical and predictable way, like, oh, it could be any of 10 things wrong with the Kubernetes cluster that it can go and look at. I like those as initial places to start automating. The higher up the stack you go and the more rare a decision becomes, the more difficult it is to augment directly with some type of autonomous agent. So a problem that happens every day is a much better problem to try to augment with an agent than a problem that happens once every six months or once a year, if that makes sense. Next question I think is really interesting. How do you isolate controlling agents from working agents? I think that this is an interesting question in that kind of what we're well, in a way, what we're asking about here is how do I isolate context windows. Right? So if I if I'm the boss agent, I have a context window and I need the information back from those sub agents so that I can decide whether to move forward or backward. But the sub agents don't necessarily need to be polluted with the entire problem. This is sort of, you know, very similar to how a human manager might work. Right? A human manager doesn't necessarily divulge all of the problems they're dealing with to every single one of their direct reports. Instead, they ask the direct reports to do specific things and then get the feedback and and move forward from there. Certainly by ensuring agentic isolation, and by setting up those structured visual agents that I was showing earlier where you deterministically decide exactly what your controller agent passes to the sub agents each time. So I try to have basically my controller agent, whenever it's asking a sub agent to do something, it fills out basically a contract. It says, hey, this is my expectation for you. This is the information I think you need. Here you go. And this is the exact format that I'm expecting you to return the information back to me in. I find that to be a very valuable way of structuring these things. And again, having that SVA that can deterministically check, hey. It did give the information back in the way that it was expected. Or, hey. The the controller agent did pass the information forward in the right way, ends up being very useful. Yeah. I we have one minute. I see a. couple of questions. I the cost, how much this cost, just be aware that we are leveraging the generative AI ecosystem. So you can answer many questions with fully open weighted, open source, or you can pay a lot of tokens. I think the importance there is being able to evaluate and experiment and provide the opportunity to that domain expert to define which LLM before going into a production environment and to experiment between one and the other one. Because cost is going to increase importance, especially when everybody is asking questions and using the agents as another Google search. We have seen that a lot too. So be aware of that. You need to be able to compare one versus the other one. Oversee how much tokens are being used by different business units. Being able to cap business units. And all of that is because of the logs that we track every time that we answer a question using the generative AI ecosystem. I'm being very aware of the time. It's 02:00 for Central Time. I really thank everybody for joining. We will follow-up with an email in terms of, thank you for participating if you wanna get in contact with us as well. Take the picture of the QR note. If you are gonna be in September in New York, that will be awesome. It will be awesome to answer any other questions in in a three-dimensional environment, enjoying this human aspect that we have is still within us. So it was great to having you. Thank you, everybody. Thank you for questions. We'll see you next time, next webinar. Thank you. Thanks for the questions. Bye,