Video: Mythbusters: AI Agents Edition | Duration: 3168s | Summary: Mythbusters: AI Agents Edition | Chapters: Welcome and Introduction (5.92s), AI Agents vs Chatbots (108.604996s), Beyond Chat Interfaces (215.475s), AI vs RPA (372.18s), Combining RPA and AI (494.47s), Agent Building Realities (840.23s), Deploying AI Responsibly (1598.175s), Building User Trust (1772.945s), Adoption and Value (1839.77s), AI Implementation Strategies (2027.005s), Measuring AI ROI (2328.26s)
Transcript for "Mythbusters: AI Agents Edition": Hi, everyone. Good morning. Good afternoon. Thank you so much for joining today's session. I see people from all over. We have several people from California. It's very early there. So thanks for coming on from Vancouver, from Toronto, from Connecticut, from The Philippines, all over the place. Berlin, Texas. Hi. Hi, everyone. Very happy to have you here. We have a packed agenda, so I'm going to dive right in. Before I do that, this session is being recorded. I already saw some questions about that in the chat, so we will email it to you probably within two days, after this event. So you'll have a copy of it, as well as anyone who obviously couldn't make it today will get a recording, or if you have to leave for whatever reason. So, yes, it will be recorded. And with that, I will kick us off. So first, I I'm Lynn from Dataiku. I'll be moderating this session today, and I'm very happy to have here with me, Emma Irwin, who's our one of our directors of sales engineering here at Dataiku. Hi, Emma Irwin. And Rowan Curran, who is, a now a principal analyst at Forrester with a focus on data science, machine learning, and AI, and we're super happy as well to have you here, Rowan. Thank you. And, of course, the topic for the day is Mythbusters. And we have chosen, several myths for you that we're going to go through today, and Emma Irwin and Rowan are going to be debating adding their 2¢ in. Definitely feel free to ask questions in the chat. I'm gonna have some timing to try to keep us moving to make sure we don't that we get through all of the myths, but we will have some time for questions. So we'll definitely be able to get to at least some of those at the end. So without further ado, let's get started with our first myth of the day, which is that AI agents are just chatbots. Emma, do you wanna kick us off on that one? Yeah. I'm happy to. I think that this is probably the most common myth that I've heard when trying to discuss, AI agents with prospects and clients and even as they become more popular amongst everyone else, friends, etcetera. And it's really easy to understand why. I think because it's comparing to something that we already know. A chatbot is something that we're familiar with and we have a good understanding of. But when you use a chatbot, obviously, you ask a question, it gives you an answer. It's a fairly simple process even if behind the scenes, it's not. And agents just can go so much further than that. It is more than an answer to a question. Of course, some of the initial steps might be the same, but we're talking multiple steps. We're talking, referencing to itself, evaluating, and actually making use of some tools, which is the kind of exciting part. Rowan, did you wanna comment as well? Yeah. Absolutely. So I think so a % agree with everything you said. And I think, you know, a lot of folks have been really focused on, okay, you know, just how can we have this, you know, way of getting information back. But really, as you said, it's really about how can we actually interact with, you know, various outside systems and take action within enterprises. But the other side of that that I think is important to, recognize, especially for folks that are, you know, building, buying, or integrating, AI agents in their organization is that it's also not just about the interface itself. So I think, you know, we can think of this as, you know, AI agents don't equal chatbots in the same way that, you know, a, you know, the early Amazon.com did not equal, like, a GUI point and click interface. Right? It was like the way that you got to the service was that, but that wasn't actually the service itself. And so when I'm looking at enterprises that are, you know, starting to, you know, integrate and implement their own AI agents, there is some focus on, you know, how can we build, you know, better chat experience or a better end user experience. But a lot of these things are starting to say, how can I, you know, incorporate these into a a workflow or augment something that is going on behind the scenes rather than, you know, have it be necessarily a chat based interface to an end user? I think that's just a really a core part of this that, is really under recognized by a lot of folks. And I think that all comes from, you know, large language models having driven the excitement in the current phase right now. But I think, you know, that's I know I said it twice already, but I'll just reemphasize it's really not just about that chat interface. There's different modalities of interacting with AI agents, whether that be, you know, a, traditional, data visualization or maybe it's just a notification or something like that. It doesn't necessarily have to be a chat interface, per se. Yeah. And I think that's really important as well when you think about, like, the full spectrum of maturity for agents. Of course, if you consider, like, the first stage people were getting familiar with is that chatbot because, like you mentioned, it's comfortable for people to use, and so you're answering questions by retrieving information. But now we want to start to go, like, one, two, three layers beyond it. Right? Enable them with tools so that an agent's actually making a decision itself on what tools that have been specified by the creators that can fulfill specific objectives. So maybe it's sending the notification like you mentioned or it's searching the web or it's doing a database lookup. Maybe it's calling to a predictive model that has already been built by someone. If they get those sorted out, they can go, like, a layer deeper, figure out how to build in multitasking agents. So, actually, like, dynamic decisions about what actions to take rather than just from that toolkit that they've been given. And then, of course, there's, like, the very, like, catchy or maybe, most exciting and scary option where people are thinking about, like, fully autonomous agentic components in their workflows, which, you know, we'd love to hear if people are starting to imagine in the chat, but that is kind of like the the future. Totally resolve that. Yeah. So I'm hearing potentially spectrum of maturity chatbots is part of it, but not all of it. It's sort of like the square rectangle. Like, chat chatbots can be agents, but not agents are not only chatbots or that analogy. Amazing. We have so many more to cover. I think unless you guys have anything else to add, we're ready to move on to number two? Mhmm. I'm ready to get into the next one. Yeah. Mhmm. Gets a little spicier. This is a bit on the same vein, but I've heard this a lot, which is AI agents are just another next gen RPA or another flavor of automation. They're kind of one and the same. I think you guys touched a little bit on this, but let's talk about this one a bit more specifically. Are AI agents just next gen RPA? And I'm happy to tick this one off. Yeah. Go ahead. Great. Yeah. So, so in some ways, you know, we can sort of think of it as, you know, the next, generation of automation in the same way that, you know, conceptually, these all these things are all progressing in the sort of same direction. But when you look at Adjenic AI, specifically, really, we're talking about incorporating, you know, deterministic, RPA and kind of task automation flows with semi deterministic, you know, decision tree, types of workflows as well with foundation models to build more abstract and complex plans to solve kind of broader business problems. And so when we look at kind of the RPA space, you know, we see a lot of the, RPI RPA vendors and some of the adjacent adjacent vendors and the, you know, infrastructure and platforms of service spaces, you know, really trying to get their heads around agents. And I think what's important to recognize here is that, you know, within the RPA space, you know, there's a couple of components that just aren't yet there in the agentic space today. So in RPA, you have, you know, various ways to evaluate, validate, and test the way that your RPA bots are working, where stay with AI agents, you know, there's a lot of, human driven evaluation and testing that has to go into aligning them to your business needs. And so that can make them a bit harder to, control, a bit harder to apply to, you know, a a mission critical business problem that, you know, has an entirely deterministic code you might have be okay with having an RPA bot delivered for. Now, the additional piece on top of this is that, AgenTek AI can also, reference additional context and information that traditional RPA agents, might not be able to address. So, in, you know, having a foundation model build that plan, you also have it calling out to external data sources to, further enhance the, choices and plan that it's building. And so a lot of people are talking about this in terms of, like, you know, short and long term memory in the agents. Really, we're just talking about, you know, how do we cache and store this either, you know, in memory or, you know, in the system storage or whatever it is. And so, being able to, further contextualize the decisions we're making, with, you know, current market information, more, unstructured information about the customer and stuff like that will, enable RPA to or enable the next generation of RPA like tools to be more flexible and to have, more adaptability within their range of choices. And now, again, there still is a lot of work that has to be done on the governance and the management side of that in order to make that, extremely reliable. But, really, it just opens up a a broader set of capabilities for, these automation, workflows. And, Emma, I'd love to hear what you all are saying too. Yeah. I totally agree with everything that you said. And for some of the people in the chat who, reminded us that some are not as familiar with RPA. I'll just bring it back one, beat and say, generally, RPA systems are software that automate rule based and repetitive tasks. So the main thing to think about with them is that they are often imitating the interaction that a human would take with a system. So, some form of, like, clicks set of clicks that a human, would normally be doing, which are always the same and always looking for the same outcome, is kind of the the the main piece of that. They require really strict, level of instruction to get work done, which is often the hard part of adapting them because you have to be able to explain. You know, it's like the story of having a kid explain how to, make a peanut butter sandwich. They say, like, take the peanut butter and put it on the bread. They just take a peanut butter jar and put it on top of the bread. It's not exactly what you meant. So defining those is, like, the hard part, which is why I think the way that I like to sort of imagine these as separate is, like, when do you choose which one? When do you choose to use RPA, and when would, like, an agent make more sense? And so for RPA, if you are looking at tasks that need to be automated, that are high volume, that are, like I mentioned, rule based and really structured, if the process requires interacting with multiple legacy systems, especially ones that don't have an API, because often RPA bots are interacting with the system, like I mentioned, through the UI, like, through making how a human does things, copying and pasting data, navigating screens, doing those clicks of the button. And at this point, you know, because we've all heard of hallucinations when it comes to some of these Gen AI tasks, if you are absolutely critically requiring consistency and accuracy, then RPA is still probably the right choice. So that's, you know, high volume structured, interacting with legacy systems, and that consistency is, like, key, then RPA is probably still gonna be an amazing choice for you to do that. Whereas when we start to imagine creating agents, why I think we are right to be excited and why this isn't just like a hype cycle of, of the future tech coming in. It's it's really about making decisions based on context, having a creative element that's being automated. And especially when you have systems that need to handle a ton of unstructured data and generate responses dynamically, this is where the capabilities of these, like, large language models really shine. And it also allows us to work on processes that require understanding or responding in natural language and with a ton of systems through APIs that that would otherwise have been, unsustainable. So I think the even more exciting piece and, like, why I really do think these are separate components to consider is that they can be combined. Right? I think you mentioned it, Rowan, is like, these are tools in your tool belt. So if you have an RPA bot that collects and structures data and it passes it to an agent for analysis and for content generation, it can be, even more powerful than if either of these things were were working independently. So it's a big game changer. And just to give a quick example of that, I think one of the places that we're starting to see this happen, you know, in kind of very early stages, I would say, is in some HR systems where, now you have the ability to, you know, through a for a chat based interface or some of their natural language interface, request your time off or otherwise do, you know, simple, tasks within that HR system. And those things, you know, have to be, you know, deterministic, get you the exact, you know, amount of hours that you are allowed to, you know, take for, you know, your paid time off or whatever it is. But then that you know, they combine that with the, you know, somewhat nondeterminism of the, natural language interaction, the intent understanding, and stuff like that. So I think, you know, when organizations are thinking about how to conceptualize and how to start incorporating the ideas of agenda AI in, as we were talking about, you can really just layer it for not just, but you can think about layering it on top of existing workflows and, RPA task automation processes rather than trying to reinvent everything from scratch at day one. Yeah. Amazing. I think that speaks to, some Mark in the comments who said it's it's not an either or, which is true. You're talking about building and layering, which I think is definitely definitely the case. In the interest of time, let's move on because I know there's a lot beefier beefier subjects to come. This one is one of my favorites because I think we see it a lot and we see kind of maybe some oh, I'm just gonna go and use the fanciest thing available when something simpler simpler would do. And so this myth is that AI agents will or should be used to solve any type of problem. And maybe, Emma, you can kick us off on this one. Yeah. This is such an interesting one because I think that the answer is a weird, nebulous maybe. Like, I think the coolest thing that I've seen come out of AI agents coming popular, amongst so many people and industries, etcetera, is that it's really done what no other form of, you know, data science or AI has before. It's hit at a consumer level first. So it's introduced to people in, like, their home and personal lives and changed the way that they think about using AI before it's being forced on them at work, which results in, like, a change of thinking that feels a bit more organic. So I can totally understand why people are coming into this going, oh, well, an agent should or could or will, do this problem for me. Right? You know, recently, Dataiku hosted, an event in New York called Everyday AI, and we discussed this idea of augmenting certain roles. And one of the roles we talked about, which had the word agent in the title is real estate agent. You know, could we just turn that into, like, an actual agent, an AI agent? And it's right there in the name. What what we did is, like, we asked the room and we asked the people who we are with, what are the tasks that a real estate agent does right now that could theoretically be handled by an agent? And they came up with tons of things. They're like, oh, an a real estate agent schedules viewings, they prospect into clients, they review applications, calculate and set prices of the homes and the rentals. They negotiate. They handle contracts. They create listings. And it's like, yeah. They do all of those things. But why do you think that they could be done by an AI agent now? It's because you have this, like, more optimistic and sort of excited view of what is possible. But but a lot of those things okay. Maybe they could be AgenTek. They don't have to be. Right? In fact, many of those things, like, if I want to schedule a viewing, theoretically, I could call an API endpoint that is related to my calendar and the other person's calendar. Right? If I want to calculate or set the price, I could use traditional ML, which has been around and is now, like, the the boring version even though it's still so powerful. And maybe the negotiation is truly agentic. Right? That would be cool to have a negotiation thought that sits beside me and helps me. But I I think why this myth is so prevalent is is because of the way that that room reacted is that it's this idea of thinking of an agent as this personal helper, that allows you to be much more creative with the ideas for what it could do to actually help you and what it can can contribute to, like, your daily life. Like, you really view it as something that's not just math and it's not outside of, like, your realm of possibility. And, Rowan, probably you're seeing something similar, but would love to hear. Yeah. I think so what I see here is that, you know, there's, you know, lots of excitement and a lot of intensity around, Atento Gaia overall. But one of the drivers behind this myth as I see it is not just the excitement, but the way that the excitement is kind of playing out in the developer space right now. So for example, MCP, the model context protocol is, you know, I'm sure most of the people or many of the people on the, the webinar, you know, have heard about this. It's essentially this way of, managing tool use, for a single agent, within, you know, single ecosystem. And tons and tons and tons of organizations have started to integrate that into their product, into their offering. But, you know, when we look at MCP from an enterprise perspective, there's still some holes that it has, especially around evaluation, governance, and some things like that that make it a little bit challenging to deploy at scale in an enterprise, particularly if we're thinking about a, you know, a regulated or a high compliance scenario. And so there's lots and lots of, you know, individual developers who have managed to, like, do all kinds of, like, cool home automation, you know, change the way that they actually do their development work and stuff like that. But it's all operating within, you know, relatively constrained, sets of, goals that they're pursuing and relatively constrained sets of, data sources and tools that they're accessing. And so there's lots of excitement around what people have been able to build in the hobbyist and individual user space because they have been able to build it there, but we haven't necessarily seen that, be able to be translated to the enterprise, in, you know, in in as many robust, resilient, and broad based applications. So, today, there's been a lot of need to, to your point, think about, you know, what is the best place that I could use an agenda AI system and what places might I wanna use, you know, a very, you know, deterministic, you know, analytical calculation or a more traditional ML score, you know, around a customer's propensity or something like that. That I think is, you know, is a a very, you know, it's the answer to that question ends up being very nuanced to organizations, but it's something that organizations should be asking themselves about how they can, you know, solve these problems not necessarily in a genetic way. And, you know, all that is to, you know, avoid some of the problems with, you know, the risk and reliability as I was, motioning toward. But, also, this can really help actually make your ultimate agentic architecture, better. So right now, I think a lot of people are experiencing some degree of agentic FOMO and seeing, like, a lot of, you know, platforms capabilities come out. But this is a space, you know, that is evolving very, very quickly. You know, you all look at this, you know, all the time I spend, you know, my whole day looking at this, and it's very hard even for us to keep up with all of the advancements that are happening here. And so for the average average enterprise, I can't imagine how hard that is. And so, you know, it's important to, you know, look at agents, think about how you can integrate them, but, you know, don't be worried that your competitors are gonna have an agent running everything tomorrow or even, you know, in six months, maybe even next year. There's still substantial work that has to happen around that and other ways to solve these problems rather than just an agentic interaction. Yeah. And maybe one last comment. I know, Lynn, you're gonna keep us on time. Is, there's a lot okay. Messages in the chat, which I think really hit the nail on the head. It's like, why do you, you know, consider, traditional ML or other tasks as not the same as Agentic? And I think that the important overview is that it actually all feeds into what is possible. Like, an agent is only as powerful as the tools that it has access to, and those tools will be built within the frameworks you start to develop now or maybe already have. Right? So the ability to create ML models, the ability to, reach out to other systems, and there's, a whole host of other actions that you may want it to take. And, of course, like you mentioned, Rowan, they're gonna have to be governed in a way. They're gonna have to be reviewed and the risk assessed. So making sure that you have those, like, foundational building blocks becomes so important. And then you can also decide if from a cost and a complexity standpoint, it actually makes sense that an agent's doing it or if, like, the building block itself was all was needed, to actually accomplish the task. And if so, that's great. You know, we have got a we've now used, like, the kind of creativity that comes along with envisioning your little agent helper to to get to the heart of the problem. Great. Yes. Let's move on again in the interest of time because I know we have a lot more to discuss. And I think we it's interesting because there's actually some chatter in the chat right now about, like, code code free. I think this speaks very well to our next topic, which is this myth that everyone in your organization will build agents. Is that true? Is that not true? If not, who is going to who is gonna be building these agents? And maybe Rowan on this one will start with you and what you've seen. So, it's definitely a myth. And I think, you know, there's both technical and organizational reasons that this will remain a myth and also should remain a myth even if some organizations don't necessarily instantiate that. So I'll start with the organizational side of this because I think I can handle it quicker. So the organizational side of this, you know, if we think about everybody building agents at our at our organization, whether that is a single department or entire enterprise, that is a nightmare, an absolute nightmare for governance, admin, and all that type of stuff. So already today, when we see folks deploying kind of broad based, you know, retrieval of meta generation, assistant copilot type systems, you know, there's a real urge so I can build, you know, the exact flow that works for me, for my particular use case, whatever it is. Now that can be very helpful for me, Rowan, as an individual, but if there's not a very clear way to, you know, see into, you know, what are the best tools used across my organization. You know? What are the tools that, you know, Emma and Lynn came up with last week that might actually serve the need that I'm trying to build right now, but I don't know about? And so I think, there's, you know, some real concern around, organizations that are trying to, you know, really push hard on everybody everywhere builds an AI agent. I think that's going too far, but I do think that we are seeing, you know, push towards, and I think a right push towards a democratization of and I hate using that word in this context, but a a broader, capability of building and integrating these agenda capabilities for more people in the organization. And what we've started to see here, you know, starting off even, you know, in the gender of AI application building space is that there's a, greater need for collaboration and cooperation between data science teams, application teams, and also the, business, side as well. Because unlike traditional applications where you can, you know, pass off some of the, you know, especially the quality control, the evaluation, things like that more to a technical team with a set of parameters, here in an agentic context, you really need those, business subject matter experts to guide the development app of the application as well as to, you know, really drive the evaluation and the honing in of that, agent's behavior. And so in the future, you know, it remains to be seen exactly what these, you know, teams look like, but I expect to see, you know, more hybrid teams of app dev, data science, and the business, but those teams having also more of an, operational role. So, kind of the dev ops team bleeding back into this, you know, more unified app dev data science team as well. Though I think that latter part will take longer. I'm already seeing the app dev and data science kinda come together more so, but I think it'll take a bit of time for more of the DevOps to come into this, in a deep way. But I'm curious about what y'all are seeing. Yeah. I think that's exactly right. It's really a question of control and capability and impact to the business. So to your point, could subject matter experts or business people or nontechnical employees be empowered to build an AI agent independently? Sure. But is that the best use of their time, or should they contribute, like you said, by lending their expertise to, like, a cross functional team so that people who have the skills to not only develop, but to push into a production environment and to continuously monitor, to manage, and and to be accountable for the end outcome, like, as a collective. I think that is, like, a much stronger sentiment. And then also there's this idea of who's responsible for making sure that it becomes widely adopted, which I think we all know is, like, in any type of rollout, you have people, you have process, you know, technology. And, like, unless people actually start to use it, like, these will remain little science projects or or, like, examples, POCs, whatever you wanna call them. Right? And so that's such a big element as well. And so, you know, everyone in your organization will build agents. Oh, maybe in their free time, they'll build something fun and rinky dink, but, like, probably not for the enterprise. Right? But will everyone use them? Hopefully, yes. That's sort of the goal. Right? That this becomes so easy to adopt that they don't even really have to think about whether it's an agent or not. It's something that they trust, that's easy to use, and that's embedded into their daily life so that they're actually getting the value of this, like, new technology without having to be the one that's that's built it. And I think three immediate areas of concern that we're seeing when customers and prospective customers are thinking about, like, putting this technology into the hands of potential users is, you know, connecting to, data and infrastructure and other enterprise systems. So a fragile or nonexistent connection to these things makes it really hard to actually, push something that could be used by the masses. The second piece is the actual ability to diagnose performance. So I guess that's first setting some kind of threshold or performance metric, who agrees that the agent is performing well, what is well in the definition of this use case, and then who actually make sure that it continues into the future. If it breaks or behaves unexpectedly, how do you diagnose that? How do you refine it? Who's held accountable? And that last piece is, like, honestly, the biggest one is when something goes wrong with an agent who is responsible both for the impact that it potentially has on an employee, a customer, a business, etcetera, but also, like, to actually make a fix. And so there's the questions you have to ask yourself like, okay. If everyone was building agents or even if a small group is, can we actually trust them to act somewhat autonomously? Like, what's the level of risk that we're willing to accept? And then from an end user perspective, if I start to use them in my daily role, like, do I understand how to use it best, and do I trust the steps that it's taking behind the scenes? If it makes a mistake and I follow through with it, like, do I get fired as the person who is able to be fired rather than, like, the agent which has no job, you know, takes no salary? And then as a business leader, can I trust that it won't go off the rails? Will it damage my company's reputation? Will it make costly mistakes? Like, we don't have to tell you guys here about, like, all of the the kind of, like, newspaper headline stories of people introducing generative AI in ways that, like, comes under question as it impacts their customers. And so I think having those three things, like the data access, the ability to identify and diagnose failure, and then the accountability concerns are really things that should be thought about in those cross functional teams that you mentioned at the top and which I think will take everyone a step back from this idea that everyone should or will be building agents to rather how can we deploy them, in a safe and a secure in a controlled way. I think I just want to add something really quick to that, Chris, if that's okay. It's because there's something important there. So one thing I just wanted to add a little onto is the trust angle of this. So I think especially because everyone's very excited about agents, really wants to roll them out. The trust angle, not in terms of, like, you know, you know, overall, like, is this a good output or not, but in terms of, do I have trust in this application broadly to say, okay. This is how I'm gonna solve my problem tomorrow. So this is the idea of, you know, like, rolling out agents broadly that you have actually tested and evaluated, not just so they give the right answer, but so you actually have users that want to use it on a consistent basis. Because, you know, most of us, we're gonna take the path of least resistance in our work. And, you know, if the tool doesn't actually do what we need it to do, we're gonna say, alright. Well, use whatever knowledge management tool I was using, you know, last week that, you know, was slower and clunky, but at least it got me to the exact answer. So, really, you know, thinking about how you are establishing, building, and thinking about potentially repairing trust with end users around agents, especially in these early stages, I think is an important aspect of, being successful. Yeah. What I was gonna add is that I think part of this myth is is maybe around also the type of use case. Like, what's the harm if I have people, you know, building agents left and right, if it's really just productivity stuff and maybe low risk and maybe I don't care. But what's interesting is we had someone in the audience ask a question, which I've now lost because it's been scrolled past. But it was something along the lines of, if I'm a CEO, how am I supposed to think about building agents that actually bring value? And maybe that's the key here as well is, like, there's a difference between agents that are giving you a 10% boost in ROI versus ones that are really giving you, like, a a 10 x boost, and maybe the people building those aren't the same. Or, do you guys have any any thoughts on that? Yeah. Definitely. I and I think that comes back to the point that we both sort of circled around, but maybe you should hammer home is that it's not about who's building them, it's about who's adopting them. Because we've all seen variety of technology come out in the past. You know? I, got new dashboarding software. I've built a thousand dashboards. No one's looked at them. Like, I have the ability to create ML models. I have the ability whatever the thing is. Right? If it's not adopted by a broader population, there's no inherent value. And so that piece, like, getting people on board also requires the trust piece that Rowan just said because I have to believe the numbers that come out of this or the action that is taken is the right one. And so there is an enablement component. There is an adoption play. There's this, like, change management that becomes so important that not just, like, an early adopter or, you know, individual will want it the whole team, the whole company culture kind of changes. And that requires it not only working today, right, having the right information, performing the right actions, but into the future as well, and being able to track that value, which I think is a myth that we're gonna talk about later. So maybe everyone hold on to your excitement. We'll talk about, like, how do you actually measure measure that value that comes out. Yeah. Maybe that's a good cue, Rowan, unless you have any last minute thoughts to to move to the next. Because I know we're gonna discuss some of these things in the next minute or two. I think what I would say is to echo something else that we were saying before that, you know, don't treat, you know, AgenDic AI as your infinity hammer. That's not a real thing. But as your hammer for infinite infinite jobs, you know, there are other tools that are still available. And, you know, really when you're putting together a complex enterprise system, you're gonna be using multiple tools to put that together. So I think, you know, for leaders trying to think about, okay, where can agents come into my organization? I think, you know, it's really more of a question of what are the, you know, problems and opportunities in my organization, and then how do we actually address those? And that could be ML. It could be, you know, using a large language model, just foundation stuff for, like, content generation or maybe it's using a more agentic flow to do something more complex to augment what a human is doing. But, yeah, thinking about it in that more specific way. Yeah. Love that. Tools in the toolbox. Okay. Let's move to the next, which is, for any company that wants to do AI, agents are where you should start. I think I started with Rowan last time, so maybe I'll pass to Emma to kick us off on this one. Yeah. Of course. I think, like anything, there's this crawl, walk, run, analogy that can be used. Right? So are AI agents the starting point for a company that wants to, quote, unquote, do AI? Well, perhaps. But it comes back to this idea we've touched on previously of it's more about getting people to think about the processes that they're already doing and also the things that they wish that they could accomplish or the way that they could, handle a problem and seeing if there's a fit for a variety of the tools that build up an agent. And so if it's thinking in the sense of an agent that helps you with your job that brings you to the right, like, set of problems, like the the things that are causing you either the most amount of pain or holding you back, then that's great. But it doesn't necessarily mean that in uncovering the use case and actually scoping it out that it will be agentic by the end. So I think it's a good place to get people thinking, but not necessarily where it will end up. And they can serve almost like a bridge to more sophisticated AI because it'll introduce basic concepts. It will also help you to imagine automation or decision support. And they're, again, becoming familiar to people in, like, a personal context so that they can actually imagine how it impacts their life, whereas something like data science or machine learning felt a bit removed from business people. If we start to think about, like, where do people start to use agents, it's in kind of three main categories, like process automation, which we've touched on briefly when we compared it to RPA, worker augmentation, and then intelligent business chains. So, actually, like, designing new ways to auto operate or operate more intelligently or drive new revenue streams. So I think as long as you're thinking within the things that have a business impact, whether it's agentic or not, it's like a good framework to begin reviewing. Rowan, I'll throw it to you. Awesome. And sorry. My Internet may have dropped there for a second. People saw me disappear for a brief moment. But so we really, when we look at, you know, the Agenca AI space, we have to think about it, you know, contextually with every other excitement that we've had in the analytics space writ large over the past twelve, fifteen years. And that is, you know, barring everything else, you have to get your data figured out in order to be successful with this. And I would say when we're thinking about Adjentic, you know, we've seen folks still have, you know, a lot of struggles getting their data in good shape for even, you know, creating embeddings, doing vectorization, and doing retrieval augmented generation. So for, most companies, I I would say, you know, having a focus on, you know, how do we get our, you know, data systems and tools, you know, prepared in order for doing, you know, knowledge serving, knowledge retrieval rather than trying to do this workflow automation stuff right off the bat would be, you know, a, a better way to approach this. But then there's also another angle that we haven't talked about quite as much that I'll just quickly reference is the kind of buy side of this. So organizations, if you do have, you know, your, you know, a specific, like, set of data, whether it's, you know, sales or customer data in a specific well curated space, then that might be a place where you could say, alright. I can buy a very well curated, very, you know, tightly, constrained agentic AI system from a company that is just going to work on this data for this purpose. And I would say that, you know, could potentially be a place to start, you know, earlier on than agentic, than in other places. But overall, I would say, you know, starting with some type of Agentic, you know, approach at the core is definitely not the place to start, especially if you're just getting started in AI. There's so many, internal skills and, you know, capabilities to develop around, you know, understanding, you know, the data structures overall, you know, machine learning, itself, you know, at its core a bit more, understanding, you know, more basic analytical approaches before you actually kind of get into, you know, more of the advanced, agentic stuff. So I I see all of this, you know, is in very much like an evolution, both towards the, you know, current state and future state as well as, you know, the present and the future. And I would say, that for most organizations, you're trying to jump immediately to Agenca AI would be like trying to jump immediately to, you know, advanced, you know, computer vision or segmentation algorithms in, like, you know, 2015 where, you know, you could go out and get those, but it wasn't necessarily what you wanted to do kinda right off the bat. And, anything else you wanna fire back with, or should we move on to the, I think, last and final? I'm ready for the next one. Let's do it. This one is on the sort of return on investment theme that we hinted at a couple of times throughout this chat, but I think it's pretty obvious that this one is a myth. Nothing is straightforward today. Let's be real. And this one is measuring the cost and impact of agents will be super straightforward. Yeah. If only. Right? So I think the first thing to talk about here is the idea that you have to measure the cost and the impact. Right? It's all well and good to, you know, play around with the new technology and be interested, read about, oh, maybe run a small trial or something else. But when you want to unleash it as it seems most people do in the chat and hopefully those who are on the phone. Right. You're going to have to actually understand what the impact is and what it's going to cost you and make sure that there is, like, an theoretical and and realized ROI. Dataiku just recently run a survey with our clients and prospective clients, and we found that, you know, 88% of them planned to increase their investment in GenAI technology this year, so in 2025. But that, another, like, 80 plus percent of them reported a ton of pressure from either their c suite or their board to actually quantify what the ROI was on any of those projects. So they're like, we're ready to invest. There's going to be a cost associated with that, but, like, what are we actually getting out of it? And if we think about, like, why, I can kind of split it into, like, four main challenges for actually, quantifying what what the impact is specifically. And the first one is that it's really hard to isolate the impact of either one specific agent or even the entire generative AI, like project or initiative from the other technology that you use, the tech stack that you're a part of, the data work that has happened behind the scenes, and other things that are going on. So you do have to kind of consider that as as the first thing, which can be difficult in its of itself. The second one that, you know, I've mentioned before but I think is worth bearing is there are not really clear metrics or benchmarks for actually defining what success looks like when you integrate an agent into your environment. Like, does it have to be a percentage better at making decisions than the human workforce? Do you have the metrics on how good the human workforce is right now? I I wouldn't. And, you know, I work in the revenue org where it's actually probably one of the easiest ones to do such a thing. We know, for example, how many emails are opened. So I suppose I could start there. But then what? Is it how many meetings can be booked? Is there a metric that I would have? Does your team or teams have something that you could say specifically, the agents are performing better at? And there's also high upfront costs versus long term benefits. Like, it there was a comment in the chat which I laughed at. It's like, this idea that agents are free. You just integrate them into your process. Like, certainly not. Every every query costs money. Every, question, has a fee. Right? Like, the infrastructure behind the scenes, all of it. And so actually putting together that upfront cost, the people cost of those working on the teams, etcetera. So, understanding when's the tipping point becomes its own thing. And then the last one, which is, like, we struggle myself, like, with my team and the rest of the org, is actually very difficult to attribute the use of certain LLMs to any one specific, like, individual or team when they start to become embedded in the business process. So if we go back to the earlier point, if everyone was building their own agent, I could say, okay. Agent one is costing this much to run, and therefore, that team, like, needs to pony up the money. Right? But if the central team builds something and then, everybody adopts it, like, do I charge them internally? What what do I do? Right? So four huge things to consider, and then to figure out, well, what are the actual benefits of doing this in an agentic fashion? Generally speaking, the two ways that we see that they're improving the the workflows and having an impact is that they provide efficiency gains, which can feel sometimes that intangible to actually recognize unless you do the calculation, and then revenue generation directly. So even I, now after listing those that I had my notes to refer back to, feel that it was not straightforward. Right? Like, there's four huge barriers to get over and then, a very, like, rough calculation to understand that that it is possible. So I think all that to say, there's kind of a work that needs to be done at the beginning to make it so that you have clear business objectives you're trying to accomplish, that you're able to then experiment in the sort of cheapest or easiest fashion, and then you have a way to actually roll out and to systematize the usage and attribution of of cost associated. I did a shameless pitch in the chat, but I will do another one here that this is really what Dataiku was made for. Of course, we've been around for over ten years, and so we didn't know at the time that it was going to be agents. We were developing, deploying, monitoring, and then and tracking. Right? But we've been doing it with traditional ML, with descriptive analytics, and many other, hype cycles that have come through. And so there's tons to learn specifically about how we could help you with doing some of this, cost impact analysis, risk assessment, security, and for those who are asking, also no code related agent building for those who already get started right away. So, I'll stop my pitch now. Rowan, I'll let you go on. I because there's tons more to be said about measuring the cost and impact. Yeah. And I think I'll go in a slightly different direction since there is a lot to cover here. And, so and this is the aspect of the actual technical side of how you measure the cost of either an Agenca AI system or even, you know, a RAG system or, other types of things and then optimize from there. So, you know, most organizations, you know, you can build a, you know, retrieval augmented generation system, even one that maybe interacts with some tools, you know, execute, something within your system like, you know, publish a, you know, a draft email to a customer or, you know, as we were talking about at the beginning of launch, process in in HR. But, you know, when it comes down to, you know, saying, alright. Well, we built this application and it costs us, you know, let's say, 30¢ for every interaction, with the end user. But how do we optimize that cost so that, you know, we're not getting, you know, let's say, you know, a dollar's worth of benefit out of every call. We're getting, you know, a dollar 50 of benefit out of that. And then so in order to do that, you have to optimize various aspects of, you know, how are you actually embedding your content, how you vectorizing it, how you accessing it, what models are you using overall, you know, are you using a, one, you know, large proprietary model for all of your calls or using, say, a combination of open weight models to kind of distribute that workload? And understanding exactly how to architect, you know, those first of all, those different, you know, flows overall and meant to measure the performance against each other, is really challenging for a lot of organizations. And so I think, you know, for a lot of folks that are kinda getting started in this space, over worrying about, you know, is this, you know, definitely the most optimal, you know, model and application flow that we can have in terms of the overall cost? And instead focusing, you know, can we get to the quality point that we need, initially and then, you know, saying, alright. Well, if we can get to the quality point here, is there a way that we can do that in a cost that is more acceptable to us? And I would say, you know, this year is really when we start to see much more attention around how do we, as application and as data developers, get to a more efficient architecture to keep that cost down. But it it again, it's like something that's very challenging to measure and can be very, very sensitive, particularly when we're talking about the models, and this is where I'll cut it off as you know. If you have a prompt that is going to a large language model, for example, you know, slightly changing that prompt, that is going to it, you know, and the response is going to change the amount of tokens in and out, which thusly is going to affect the amount of compute and the amount of cost for that actual interaction. So these systems, when you have, you know, many models from together, many different workflows can be particularly sensitive to small changes in ways that, you know, more traditional applications, may not have been under a lot of circumstances. So I think that's a a really challenging aspect of the cost aspect here, particularly when we move beyond, you know, d one, d two of these applications, and everybody's really trying to get the, you know, the most juice out of the agent that they can for the lowest cost. Yeah. That's great. I I'm a any final thoughts on this one? And then maybe we can take, like, one question. We're getting to time, but I have a good one in mind if we wanna if you're ready to to move on from this ROI. Yeah. Jumping questions. So there's one question which is kind of a big one, and that's what are some good examples of use cases for Adjentic? But maybe what we can do is keep it away from, like, theoretical and and maybe, Emma, you have I threw a couple examples from our customers in the chat, but maybe, Emma, you have a, like, a a favorite real example that we've seen at Dataiku. And, Rowan, I don't know if there's any, real implemented, not theoretical examples that you've seen on your side. Maybe one example each if you have one. And, Rowan, can we start with you? Yeah. So I think, what's the best example? So I'll do two super quick ones. There's one super quick one is, software architects using agents to help them, build their, design documents and build, kind of drafts of how they're gonna build out applications. Now those tend to be agents that are tightly constrained within knowledge systems that are not so much action taking, which is honestly where we see a lot of the agents stay. They're very much focused on information retrieval and information serving to an end user rather than taking complex actions on the back end. So for the ones that are actually taking action on the back end, kind of the best examples that I've seen about today are the ones that are, you know, tied into, these systems of work. You know, again, for prefacing the example that I talked about before, the HR systems where you have, you know, more complex and ambiguous user query coming in, but it's a relatively simple set of known outputs on the other side. So the agent isn't trying to say what are the entire world of, decisions to make. You know, what is the best decision amongst these three or four different options? So tightly constrained types of, agentic scenarios are really where I'm seeing success today and the information retrieval. Love it. And, Emma, do you have a favorite? Those are two really good examples. And then I would say that favorite is maybe a weird way to describe a use case that your client's working on. But, one of the most recent ones, that we're doing is helping to deploy basically a, like, technical support agent. So, again, it already has the term agent in the job title, so it's such a great, option to to automate. Basically, the one we're working on right now is that, the customer is using, ServiceNow to sort of orchestrate all the support tickets that they get. We are connecting into that, as a knowledge base so that we could identify proper proper steps that need to happen. So let's say, you know, your client opens a support ticket. It has a kind of ambiguous question that happens or there's been an incident. They need something, from that support. It will the the goal of this agent is to understand from all historic tickets, you know, is this something that could be automatically resolved with some quick, information? So a report back to that customer of what to do or how to resolve. If so, actually being able to to respond to that, directly. So interacting with the user, just like a cool, element. Right? It's not just serving knowledge, like, internally. It's having an impact on, like, an end customer. If it is identified that actually the issue is a level two, so no longer something that could just be resolved with a simple, explanation, then it's actually able, like, take take the next step to identify what group of individuals internally does it need to escalate to. So, like, level two questions might be handled by team one, two, or three. Right? And redirecting it to the right person. It can also then flag tickets if they respond correspond to a major incident or security, impact. So if this is going to have an impact on revenue or if it's actually, like, a downtime, if it reflects security issues, etcetera, then it can also escalate those, and send internal reports. So I think it's a really nice one because these types of teams, like these support teams, are already used to some forms of automation, and now it's just the, like, level above what they were already doing. Right? It's giving them the intelligence aspect, rather than just, like, following a set of business rules where potentially before it was looking for keywords or, like, other impacts. So I'll leave it at one since I know we're running on time. Amazing. Yes. I think we're pretty much at the end of time. Thank you everyone for your questions. If there were any big ones asking for examples that we didn't answer, we'll try to include some more examples in the follow-up materials. Thank you, Rowan. Thank you, Emma, for joining and for all your insights. It's been really great. And, hopefully, we'll see you at the next webinar in this series. We'll also include in our follow-up a link to where you can sign up for future future webinars in our agent series. So thank you. Thank you so much.