Video: Mastering Orchestration for Enterprise AI Agents | Duration: 3360s | Summary: Mastering Orchestration for Enterprise AI Agents | Chapters: Welcome and Introduction (6.16s), Introducing Dataiku AgentHub (82.405s), Agent Orchestration Importance (190.255s), Orchestrating AI Agents (386.35s), Dataiku Agent Architecture (690.74s), Agent Architecture Explained (1687.79s), Agent Capabilities Overview (1920.2s), Integrating External Agents (2174.235s), Feedback and Accessibility (2510.955s), Agent Evaluation Strategies (2712.725s), Human-Driven Ground Truth (2976.135s), Future Product Vision (3127.955s), Future Webinars Announcement (3305.49s)
Transcript for "Mastering Orchestration for Enterprise AI Agents": Hello, everyone. Welcome to our orchestrating AI agents for enterprise impact webinar. We're super excited to have you here. We're just gonna wait a couple minutes for people to join, but I already see a lot of people, saying hi on the chat from multiple places, UK, California, New Jersey, Munich, Calgary. So it seems like we have a very global, community here. Welcome, everyone, and thanks for spending the time with us. Before I pass it on to our presenters, I just wanted to let everyone know that tomorrow, you'll be receiving the recording of the webinar as well as the slides, so do not worry about that. You're gonna have that for your reference to watch on demand and review the materials. And we do encourage everybody to ask questions, during the webinar. We're gonna also save some time at the end to get to answer as many questions as we can. And without further ado, I'll pass it on to our speakers so that we can get started and get into the fun stuff. Awesome. Thanks, Barbara. Hi, everyone. I'm Julia, and I'm joined by my colleague Christian today to discuss agent orchestration. So just to start, wanted to give a quick rundown of Dataiku. Dataiku is a universal AI platform that brings together your people, data experts, and business users, your data, whatever format, wherever it lives, your governance, so visibility and control across the org, covering all steps in the data to AI life cycle, and your technology. You can swap out cloud providers, data storage, or underlying models. Dataiku works as the unifying layer atop almost any tech stack. Concretely, Dataiku helps business users create everything from visualizations to gen gen AI apps and agents alongside data scientists working in the same interface to build shared understanding and ensure your teams can drive impact with data. At the same time, Dataiku ensures IT and compliance teams are able to keep track of what gets created, how data is being used, and enforce the unique quality, costs, and security requirements. Recently, Dataiku launched AgentHub. AgentHub brings all agent experimentation, distribution, and governance in one place. For business teams, that means the freedom to create and use agents tailored to their needs. And for IT, that means full oversight of access, data, models, and life cycle management. Access, data, models, and life cycle management. In this webinar, we'll walk through orchestration, why it matters, and a demo of how to orchestrate agents in Dataiku using AgentHub. So why agent orchestration matters? As a reminder of what makes agents different from Gen AI apps, so Gen AI apps to this point primarily produce content, text, images, code, and so forth from a prompt. AI agents go beyond content generation to take meaningful action, so incorporating memory, reasoning, goal setting, and adaptability. At Dataiku, we're seeing our customers leverage agents across every industry and every use case, from sales to finance and manufacturing to pharmaceuticals, for answering things like RFPs, investigating money laundering, and even clinical trials intelligence. In order to do this successfully, agents need to operate across a mix of modern legacy, and third party systems like ERP tools, SaaS platforms, custom built internal software, cloud and on prem apps and databases seamlessly. They need to understand how to connect to APIs, handle varied authentication schemes, and translate data between systems. This requires robust orchestration to ensure agents can interact without errors, maintain context, and respect compliance or data residency rules. Without orchestration, businesses run into risks like multiple agents performing the same task without awareness, wasting compute and API calls, acting on inconsistent or outdated data leading to contradicting outcomes, inconsistent access controls exposing sensitive data, inefficient queuing from lack of task prioritization, simultaneous uncoordinated queries, overwhelming APIs, and the inability to trace, understand, and debug agent actions. Effective orchestration is what allows agents to operate together efficiently, safely, and at scale. There are four key components that make this work. So task coordination, orchestration decides what happens when, managing dependencies and timing, so every task runs in the right order, agent delegation, which is orchestration assigning tasks to the right agent based on the context, data access, and permissions, ensuring the best fit for each job, monitoring and optimizing. As agents work, orchestration tracks their performance and adapts on the fly, pausing, rerouting, or adjusting when needing when when needed to keep things running smoothly and efficiently and control at scale. So orchestration lets IT enforce policies, manage access, and maintain trust and traceability as the number of agents grows. With those building blocks, coordination, delegation, monitoring, and control in mind, I'll hand it over to Christian so we can see what this actually looks like in action with orchestration in practice. Perfect. Yeah. Thank you very much, Judah. And, hello everyone. My name is Chris. I lead our Gen AI Field Specialist here at Dataiku. And today I'm really excited to run you through, a couple of slides and then, we'll dive straight into a demo to, to really bring this topic to life. Really two key trends that I'm seeing when working with customers and, of course, in the market as well. The first one, of course, is agents are everywhere and I don't think this is going to change anytime soon. We're seeing lots of different, organizations starting to release specialized agents, which are usually, specialized on on their specific platform. Obviously, these these can be really, really beneficial, very performant at performing specific tasks on on those platforms. I think this is a good thing. I, I think this is only going to continue and, organization will be able to really leverage these if they manage to be able to orchestrate all of these agents in a in a unified way. The next trend that I'm seeing is really the single agent use cases, which a lot of organizations have already started experimenting with and even putting into production, moving towards multi agent use cases where suddenly we're looking at various different agentic architectures, where we've got supervisory agents with specialized agents under the hood. There's a couple of pros and cons for each. What we see is specifically with single agent use cases, once the certain toolset grows, it can become quite complex for the agent to manage all of those different tools. And so this is where it's a really good point to start to experiment with multi agent systems where we actually delegate specific tasks and each individual agent can have specific instructions and also its own LLM, obviously to to leverage as well. So we can really, suboptimize each individual agent and, of course, have that supervisory agent that is delegating those tasks to the specialist there. So when I think of orchestrating agents, I obviously think of agent building, right, prompt engineering. But more importantly, I start to think about the systems that we want our agents to interact with. Agents are only powerful because of the tools that they can leverage. Without it, it's it's just an LLM. And so we have to start thinking about, well, what are the the types of tools that I want my agents to be able to, to interact with and leverage to essentially make those autonomous decisions and and become, the most powerful version of themselves. And so, I also start to think of these in in topics. Right? We've got structured data, which can live across my organization in various different databases, unstructured, or Blob storage databases like s three Blob storage on Azure, and AWS Cloud storage as well, different SQL databases that we may have, Snowflake, Databricks, and, of course, potentially some legacy technologies as well, which we often see, within enterprise organizations, Hadoop, etcetera. And so we have to start thinking, okay. Well, how do I enable my agent to start interacting with those underlying data systems, in in the most effective way possible. The next is obviously unstructured data. Right? We've got unstructured data in PDF, text files, word documents, spreadsheets that we may have. And all of these live, across our organization as well. So how can we effectively start vectorizing, this data into our preferred vector database and enable our agents to interact with that? Data cleaning obviously plays a big role into this. Like, how do we prepare our data to be consumed by agents? Sometimes we may wanna massage that data in a in a specific way to make it as optimal as possible for our agent to be able to interact with that dataset. And then we get into traditional machine learning models. Right? We wanna be able to use the best tool for the right job. An agent isn't gonna be good at being able to do time series forecasting. For that, we obviously wanna use a time series model. And so how do we make it really easy for agents to connect into those time series models, those traditional ML models, random forests, etcetera, to be able to make the right classification, and provide the user with the most relevant answer. You can quickly see how when we're thinking about agent orchestration, it's much more than just the agents. It's the underlying systems which actually become the most complex tasks. It's actually very similar to traditional data science while we were building machine learning models. Building the machine learning model is actually not the hardest part. It's all the stuff that comes before it. Interlinking all of these different systems together, these external agents into into our, Dataiku agents. Making all of that work gracefully is oftentimes the, the most complex and challenging part of building production grade, agentic systems. When we think of Dataiku, really that's the mindset, that's the philosophy. How can we unify all of this into one enterprise platform to enable you to connect to all of your data across your organization, both structured and unstructured, enable you to very easily choose the right tool for the right job, create that time series model just as easily as spinning up that agent, and interlinking and connecting all of these in a governed and controlled environment so that your organization could feel confident to pull these use cases into production. And with that being said, let's actually hop into the demo here. So I think if I stop sharing there and I share my screen, you should be able to see Dataiku. So this is Dataiku. For those of you who haven't seen it before, this is kind of a visual representation of our data pipeline. Our various different systems, of course, interacting with one another. And this is actually a use case that we, we built together with a customer of ours. And so this use case, it's a relatively simple one, but but very impactful. It's a product recommendation engine. Right? So we've got, specific financial products which, which customers are investing in today. We want to make sure that based on that historic data, we're recommending the right product to the right people when they get on the phone with us for a customer support case or if we reach out to them by e mail. We essentially want to be able to recommend them the best product. Agents can become really powerful assistance for for, for salespeople, but also for our customer success and customer support team, to be able to essentially interact with those agents and get the right information very, very quickly versus having to go into all of these different subsystems themselves to fetch this information. And so the first thing that I'll highlight when we look at this project here is that agents saw everything in pink. That's everything to do with GenAI. And so you can see that that's a part of this project, but it's certainly not the only part of this project. On the left hand side here, we've got everything related to data preparation. So you can see here in this case, we're actually pulling in from Snowflake, but this could be any database that we've got access to, in our organization. Right? So here you can see all of the different SQL databases, network databases, Cloud storages. Really any database that we need to connect with, we can pull data and then stop massaging that data and stop preparing it to be consumed by our agents. Now in this case, we've actually done a join. It's a it's a relatively simple join. Customer ID is left join. So we're joining all of our demographic data, interactive data. We're joining all of our product data all into one single dataset so that our agent has access to all of the dataset, all of the data that it needs to have to be able to provide us with the insights that we're looking for. The next section here is machine learning. And so this is where we're actually using traditional machine learning, based on our historic data to, to essentially come up with a next best product, for our user. So we can actually go into this one. It's a random forest model that we created using Dataiku's visual, visual machine learning capabilities. We can, of course, go into this decision tree because it is a forest based model and see exactly how this model is making certain decisions so that we're sure it's unbiased towards any specific age group, for example, and its performant, across the board. And so I'm very happy with this model. So this is actually a model I'm gonna start exposing to my agents to be able to, to leverage, when we're asking a question. Cool. So let's get into the agent stuff. In this case, I've actually, experimented with two types of, agent architectures. The first one is a monolithic agent, where it's just the agent connected to the tools directly, one single agent, and it's it's doing all of the tasks for us. The next one is actually a more complex system, but it might actually be, more resilient and more easy to maintain in the long run. So this is actually a multi agent system. And so you can see it's actually comprised of many different agents that are specialized at performing certain tasks. We've got a SQL lookup agent here. We've got our product lookup agent. We've got a product predictor agent, and we finally got an email agent. And all of these, of course, have their own prompts, have their own LLMs, which we'll dive into, in a second. Before we dive into the actual agents themselves, we, of course, wanna wanna start looking at tools. And tools are the most important part when we're thinking about agentic architecture. The better your tools are, the better described your tools are, the more performant your your, essentially, see your use case be. And so what we can do is we can come over to new agent tools here. And here we can see all of the different tools that are that come out of the box with Dataiku, which we can start attaching to our agents. So we've got inline Python, which is extremely versatile, enables us to to essentially allow our agents to execute Python functions, dataset lookup, knowledge bank search if we wanna start looking into, vector databases, LLM mesh query. And this is a really important one when we start thinking about agentic orchestration. This enables our supervisory agent to call upon, different sub agents to be able to, to delegate tasks essentially. Yes. We'll we'll look at these here now. So you can see all of my different tools. We'll actually go into the SQL agent tool. So I'm actually linking to the specific sub agent that I want, on this tool to connect to. So this is the SQL lookup agent that I'm connected into. I've written a very basic description, and then, of course, we can test it from this user interface as well. The other tool I'll show you here is our external model. So we've got a couple of external, agents running. Again, these might be across your organization, hosted in various different platforms or hosted yourself. Right? If you're using open source tech, you may be hosting, your own agents, as well. And so I'm using, in this case, MCP to be able to connect with these agents. And so in this case, we've got a business case analyst, which is a specialized agent, which will help us, especially come up with, or help us refine the business cases, of my organization. I may wanna delegate to these external agents. I also have a WebResearcher, which I've run which I'm hosting myself, which I wanna be able to delegate to as well. But here, you can see how we can use MCP to delegate to different tools, but also to different external agents, when we feel it's necessary. Alright. So now that we've kind of understood how we can start creating those tools and start connecting to various different systems like databases, models, and vector databases, we can actually go into, agent's hands. So we'll actually dive into the product predictor agent. We'll go into v one here. So this is our product predictor agent, and the the nice thing about having a multi agent system is that we can really specify which LLM we want each sub agent to have. And maybe you've got a really complex subsystem like SQL querying, which requires, kind of a lot of intelligence from from the agent. Well, we probably wanna assign a model like GPT five. That's a that's a very performant model. We can set it's, it's thinking time, it's reasoning time to high, and so we expect really high performance queries, SQL queries being executed, using, using a a cost model, essentially. But But for this specific task, it's actually a relatively basic task. We just wanna fetch a specific record from a data dataset named the customer ID, and then we wanna use a traditional ML model, the the random forest model, to predict the best next product. And so that's a relatively simple task. So we can actually tell this sub agent to use GPT four point zero. So that's another great advantage that you get. It enables you to really do some cost, performance optimization at a more granular level using these various different sub agents. So here we provide our instructions and then again those tools and we can add new tools to this agent as well. And then we can quickly start doing quick tests or, of course, chatting with it and running a multi turn conversation to see how it performs, its specific task. That's kind of how we set up our agents here. And, of course, our supervisory agent, has four, separate tools five separate tools, I should say. So it's got the predictor agent. It's got the product lookup agent, the email agent, the SQL agent, and then the external agents as well, which it's connected into and can start delegating past to. Okay. So now that we've seen the agent building, let's dive into how we can maybe start using these agents. So in this case, I've got two ways in which I want to leverage this agent. I wanna leverage this agent both, in a chat based context. Right? So multi term conversations in a single unified interface so that my users can interact with all of my enterprise agents, as well as building their own agents in a really easy to use, interface, which I'll show you here in a second. But we also have the concept of running agents in batch. Right? Every week, every month, depending on maybe every quarter if you want a quarterly write up. And this is actually more common than you might think. So we we natively provide the ability for you to do that. In this case, what I've actually done is I've created an editable dataset. We can come up to edit here, and here we can start editing, this specific dataset. I've only got one simple task. So I say, hey. Please take a look at our top five highest spending customers and see what products we should be recommending to them. Give me a detailed report and pitch of the products we should be recommending. So in the flow, I've just created a prompt recipe. The prompt recipe is super simple. It just has the agent, which is specified here, and then the prompt input is just the task from that specific dataset. And we can come to the output dataset here, and here we can see the detailed report that's written. The agent's gone off. It's performed its query. It's find the top products to recommend to these five customers. And of course, now we want to get that emailed to me, so I'll just use the Dataiku email tool, send me those emails. Every week, I now get a report of the top five customers and then the products that we should be recommending towards them. And you can customize this completely, set multiple tasks, etcetera. Now the next thing before we dive into the chat user interface is actually gonna be around agentic, evaluations. You want to essentially continuously monitor how your agents are performing over time. And so in this case, I've I've set up a couple of different tests to continuously test the monolithic agent versus the, supervisory agent system, the multi agent system, to see which one is performing best, but also to perform a bit of a cost analysis. So we can see performance versus cost. So in this case, you can see here I've got my evaluation question dataset. So I've got three evaluation questions. And I've also got a ground truth dataset. And ground truth is really important where you're working with SMEs to say, hey. If I were to ask you this question as a subject matter expert, what would in fact be your answer? So that we can start benchmarking our agent versus, SMEs to basically see, how well it's performing. For ones where we don't have ground truth, we can use LLM as a judge techniques where the LLM becomes a judge and provides us insight into how well it's actually performing. And so we can dive into this one. So we can go into the monolithic agent. And here we can see the answer relevancy being the core statistic here as well as cost and latency. So across three runs, the average cost per run is 0.021, which is very low. The latency is, is about sixty three seconds, which is actually pretty good, and the answer relevancy is 0.592. We can then come over to the multi agent evaluation store and start looking at the exact same, results here. We can see that the answer relevancy is actually much better. So it's 0.61. The cost is actually lower. So we've used lower cost models. As I mentioned, multi tenant agents allow us to specialize, which LLMs are more powerful for which, separate tasks. And so for a lot of the tasks, we don't need powerful LLMs. So we've been able to lower the cost and in fact lower the latency as well because, again, we're using a more simple model. And so the latency is is generally, much lower for something like GPT five Nano or GPT five Mini or 4.1 than it is for GPT five full. Right? That is a really powerful model, but, again, it's not needed for all tasks. And so we can see the advantages of having these, these multi agent systems come to life in, of course, these, these benchmarks. And over time, as users are leveraging, agents, we can continuously monitor these, these systems, and these tests as well against the conversation history that our users are having. We can do that by going over to scenarios. So if we come over to scenarios here, you can see I've I've set up two scenarios in this case. One for the weekly agent run. So this will run every week and it will just fire off those tasks for us. Right? That singular task I've got at the moment, but maybe in the future I want to add tasks. So in this case, it will run once every Sunday and the steps that we will run is super simple. It's just one step. It it, it essentially finalizes that dataset, completes the task for for us and then fires off those emails. And then the, other scenario that I've run is the continuous performance monitoring, which I've set up. This runs every two weeks because we don't need to do this every week. And the steps are a little bit more complex, but the thing that we run first is all of our evaluation stores. Next, we wanna verify the rules or checks that we've set up on those evaluation stores so we can set specific benchmarks for each individual evaluation to say, hey. If it falls below a certain criteria, then, of course, we wanna raise an alert and, and essentially, be be alerted. And so we can do that using the send message. So in this case, I'm using Microsoft Teams, but that can be email, Slack. It depends on what your team likes to use, and the message that we'll send here. And so, this will only run if some prior step is filled. So if a check is filled, this will run. It will alert me to let me know, hey. Those agents that you've got running in production, they're actually not working as well as they as they used to, and you probably wanna take a look, and start, not analyzing why they might not be performing the way they should. Alright. So now we've got that done. Let's actually start looking at how we can start exposing this in a conversational interface. And so this is Agent Hub. You Probably heard all about this already, but essentially, this is a unified, interface that enables your organization to expose all of your different enterprise agents in one single interface so that your users don't have to go to Teams for one agent, then maybe, another platform or web app for another agent, then maybe, going to a specialized SaaS platform for another agent. Really, what we're looking to do is unify all of this so that your users can log in in the morning, distribute their tasks to the various different agents, and stop performing the work that they need to do versus having to go to each individual bookmark, in, in the morning. Trust me. I used to have to do that since we've launched Agent Hub internally. We're we're using this a lot, and it's really made my life, a lot easier. So in terms of our agent library, we've got two enterprise agents. We've got that product recommendation engine. This is that multi agent system, so a supervisory agent which can delegate tasks for us. And then we've also got, an email agent. So I can use this agent to look at my email, fire off emails, do those types of tasks. We also have the concept of my agents. What we've seen happening is within organizations, we see various different individual agents spawning up. People may wanna do task in a very specific way. Like, for example, I'd like to write emails in a very specific way. But you can see here, Chris is email writer. Now come over to the My Agent edit interface. And so this is where your users in the in an end user interface can interact and build out, agents, and, in a in a very specialized way, in an individual way. We can, of course, share these with our team as well. So if I did wanna share these with, you know, two two members of my team because we all like to write emails in in the same way or because we all do a task in the same way, I can very easily do that. So in this case, I said, hey. You're my email drafter, and, the way I like to write my emails at the end is is insert a little bit of humor and, and poetry and have a nice little poem at the end related to the topic that the email, is, is being written about. Can make can make work a little bit more fun. Right? So we've then got our base model, which is GPT five Mini, but we can select pretty much any model that, that we we have access to. We can give it documents. So if I always wanted to reference specific documents, I can do that. And then we've also got the the concept of enterprise tools. So our IT team, our central team that manages Agent Hub, may say, hey. I wanna I wanna enable all of my users or a group of my users to be able to interact with various different tools in my organization. So in this case, I can actually use different agents as tools. So I've got that email agent, that enterprise agent. I can leverage that as a tool. I can also leverage the SQL agent, which is hooked up to all of our SQL databases, as a tool as well. And so really and my external agents if I wanted to. And so in this case, I'm using my email agents so that when I select this quick agent, it can also start leveraging, it's it's we're giving it the capability to delegate being able to send and receive emails as well. Finally, we can just give it an agent overview and some example queries so that I remember how to use it, when I, back to it. Alright. So let's go to some of the conversations I've been having within this interface. So just zoom out a little bit there. So the first thing that I did is, hey. Can you please look up customer ID 27 and give me the information and find out what is the best product that you'd recommend to them? And so you can see here, the agent's gone off. It's gonna fetch the information using the different tools and the sub agents that it has. We We can actually come down here and we can come over to see the details. Let's come over to activities. And here we can see that agent hub is actually distributing, a query to the right agents because I'm interacting with two different chat agents here. It actually says, well, this is definitely a task for the product recommendation agent. So it gets the agent ready, the agent starts thinking, and it starts executing some tools. So it started executing the SQL tool, and then it used the predictor agent, as well, which then came back to us with the right prediction to, for us to use. Next, I said, hey. Let's fetch the details of those products that we just, that we just predicted. And so it's gone off to fetch all of the information on these financial products that we should be recommending to customer 27, and we can actually see that again in activities. Whereas, in this case, actually gone and used the tool, the product document lookup agent, and then finally the predictor agent again to fetch me this information. Then finally, I said, okay, great. Can you direct me an email to this customer recommending this product and why? In in this case, it goes, okay. Well, this is probably not a task for the product recommendation agent anymore. It's a it's a task for Chris's email agent. Right? As you can see here, we've got those two agents. We've got my email agent hooked up here and my product recommendation agent. And we can, of course, select more agents than two, to interact with. Maybe you wanna select all of your agents and just have one chat for all of your agents. And so let's see here. So it's written the email. It's really strong in terms of detail and facts. They're quite formal. This is exactly what we wanted. And at the end there, it says, hey. A small poem to finish up because tech plus investing is inevitable poetry. So a nice little segue to, to the poem here. And here we go. This is the the poem that I wrote. We can actually come over. We can see the details. Pretty simple. It just used that agent that I, I had set up here. And finally, let's just take a look at how we would use external agents. So remember, we've got that external agent hooked up, as, as MCP. And so in this case, I'm asking, hey. Can you do a pattern analysis on our customers and data and products? So, right, I need to do some stuff with my unstructured data, with my structured data, and look into age and typical products, and then draft a short business plan and double down on those patterns. The patterns that we've got right now, they're great. We should do that more, come up with a business plan here. So in this case, we can actually see what it did. So we can come up with activities. So we'd use the SQL agent to kind of get that structured data, then it did the, vector database search. And then finally, it goes, well, hey. We should probably use the external agent because that agent is specialized in creating those business plans that's being set up by another team. So we wanna use that external agent, and then finally gets back to us and provides us some plan, which is great. Now as a developer, we may just wanna wanna get a little bit more detail into actually what's going on under the hood. And so we can view this in Trace Explorer. So Trace Explorer gives you a really good insight into how this agent is orchestrating between these different sub agents, between those external agents. So here we get a graphical overview, a tree based overview. We can come over to timeline, and then my favorite one that I use the most is Explorer. So here we can see all the different tools that you used and the details on what what what got sent, what what got returned, how many tokens it used. Here we can see the prompt and the completion. And then finally, also an overview of cost. So this entire agent run, took about $0.06, which is really good, very efficient. So I'm very happy with the with this specific agent run. That was everything I wanted to go through today. Yeah. I hope that was insightful, and, I guess we can move to, to q and a here. Awesome. Thanks, Chris. Let's see what questions people have for us today. So we can start from the bottom. So do you monitor for AI agents in use across the enterprise? If so, how? So when we're monitoring for our genetic activity right now, that's, solely if the agent is executed in Dataiku. Can, of course, import traces. If you've got a centralized trace server, we can, of course, pull pull from a dataset to to visualize those. But right now, the the main traces that we're visualizing here are all agent runs that happened in Dataiku. They can, of course, interact with external agents as tools and and that will be visible, but it's mostly for for agent runs inside of Dataiku. Matthews asked, is an agent just an LLM with tools? So an LLM is the foundational component of an agent, but the agent has more than just the LLM itself in in order to be able to act autonomously. So kind of to what Christian spoke about during his demo, there's, like, the tools, so access to APIs, databases, and other software systems. So an agent can call a calendar API, query a database, or send an email. Those are things that an LLM can't do by itself. Memory, so it has the persistent memory to remember facts, goals, or past interactions over time. LLMs are usually stateless. They kind of forget everything after the conversation ends versus an agent keeps context across sessions. There's, like, the planning and decision logic too. So control loop that lets agents decide what to do next, the goal and autonomy orientation. So instead of, like, giving a agent a prompt, like, find my find a flight for this. You can tell an agent to book a trip to Paris, and they'll be able to plan and execute steps towards that goal versus just one single step. And then they usually operate inside environment, So, like, web browsers, games, enterprise software, so they can kind of understand when a state changes and then respond dynamically. Another question, which you also kind of spoke to during your demo, but AI agents are mentioned. How would MCP servers be accounted for since they contain tool sets that agents might use? Yeah. Great question. So within data, we've got the ability to host local MCP servers. So the servers that you saw connect to various different MCP clients as well as external MCP servers, which expose tools, external agents. And then essentially the way I see MCP server is it's like a toolbox. Right? Instead of having to create each individual tool again and again, which can be quite cumbersome and complex, you can just get a toolbox. That toolbox can be a toolbox of interacting with ServiceNow, with all of our internal enterprise agents. Maybe you create your own MCP server. We can pull that MCP server, those files into Dataiku and expose them to our agents really, really efficiently. So the way I see it is it's a it's a real time saver essentially. And then Olga asked, which you also kind of covered, but do we have a tool to monitor each agent resources usage? Yeah. So each agent that is, that is being executed, you can go into Trace Explorer and see exactly the resource that it it was using. You can also centralize the statistics, of at a macro level so that you can look at the how the agent's being used across the enterprise and kind of see how many tokens are being sent to, to the agent, to kind of monitor performance, latency, cost, etcetera. And then Navesh asked, and I think this is a clarifying question about your demo, when you're building. Does the build mean we are building the system, the flow of agents, or an orchestrator agent? Alright. Can you repeat that question again? I was just read reading the chat. There's some good questions in there. Yeah. So Nubesh asked, when you're building in the beginning, were we building a system, the flow of agents, or an orchestrator agent? Pretty much all of the above. So, really, the demo and, again, that that's the thing I wanted to highlight is that when we think about agents, we just wanna go straight to the agents. Right? It's like the the cool part, the LLMs. We wanna see it it it think and reason and do its own thing. But actually, majority of the time, it's it's building out these quite boring data pipelines and and workflows. So how can we do that as effectively as possible? And then the next thing that I see a lot of organizations struggle with, is kind of that that connectivity. Right? It's like, okay. Well, we've done our data preparation over there in our old legacy database. We have a Postgres database. Then we've got, you know, a vector store over here, vector store over there. We've got unstructured data in SharePoint, but also an s three. And it becomes, yeah, really messy very quickly. And so the complexity of building these agentic systems isn't necessarily the agents. Like, everybody can can do that quite quickly. Right? You can write a prompt and you can tell it to reason and when to delegate tasks, etcetera. But it's like, how can we quickly and efficiently connect it to all of these different subsystems as tools, so it can perform a task like a human would? I think that's really the the power of Dataiku. So the demo took you through that entire thing. It's not just orchestrating agents. It's it's orchestrating what's under the hood as well. And then, Maha, so these three questions are kind of similar. So is it possible to leverage existing private OpenAI environment? Is it possible to use in Teams, and would you be able to use a self hosted LLM? All three, yes. So private open AI shouldn't be a be a problem. Dataiku enables you through the LLM mesh to connect to pretty much any LLM that you want to including, on premise LLMs. So, a lot of the organizations that we're working with are definitely experimenting with, with with local LLMs, open source models, which are actually surprisingly powerful. As I said, I think what's really interesting is seeing this divergence from going for the biggest model every single time towards more specialized small models for specific tasks. And again, as I as I said, as we're growing towards this multi agent system, being able to apply small models for specific tasks that might be locally hosted for very sensitive topics, can can be very performant. There's also, of course, the topic of fine tuning as well, which is comes in and out of trends. Some organizations I'm seeing are really heavily experimenting with fine tuning. Some have said it's probably not worth it. So that's also a topic. And then, what was the what was the second point, Julia? It was would you you mentioned self hosted LLMs. Is it possible to use it in Teams? There's some limitations with the Teams platform that can make it difficult to integrate agents. We have done it before. It it is possible. So, yeah, if that's if that's something you're interested in, we can, we we can discuss it. And then Tom asked, Chris, I'd love to hear about your personal use of agents in a daily way in your actual work, in particular, how you're using Dataiku for that. Oh, great great question. Yeah. I, I actually don't know how much I'm allowed to share, but we we use it a lot. So we have our own agent hub platform, so our own Dataiku agent hub platform, which is connected into pretty much all of the underlying systems across our organization. We have a central team, which is actually, one of the developers there is, is one of my best friends from university, joined Dataiku as well and he's actively working on these use cases, which is really cool because I get to see what he's working on a day to day basis. Essentially what they do is they go to all of these different business units inside of Dataiku and ask them essentially what would be the most impactful agentic use case for your specific division. We've kind of compiled that list and started working on those use cases. A lot of those are in production at the moment. What's really cool is being able to see how individual teams can come in and interact with that central team. For example, the solution engineering team, which is the team that I used to be in, have a very specific agent, for looking at customers essentially. Customer data, all of the customer data that we have, how can we work best with customers, what are the different products, the features that they're using, what are the different tutorials that we might recommend to them, all of this type of stuff. Solution engineering team actually worked really closely on fine tuning the prompts, specifying the different sub agents that should be part of this supervisory agent, the different systems that it needed to connect into, whether the data was clean or not, all of this type of stuff that only a subject matter expert would know. And, of course, also creating ground truth q and a. Right? So the solution engineering team came in, created q and a's to be able to say, this agent is actually performing or not before we roll it out into production. So that's one agent that I can talk about which, which we use quite a bit. But, yeah, it's, it's pretty cool. It's definitely made me a lot more efficient. And then, obviously, coding as well. Like, it's it's it's crazy how efficient these agents are at coding now. So I'm able to do a lot more much more productive than I used to be. Nice. Tom had another question. Have you actually been able to move all of your agentic AI to Dataiku? If you've had problems meeting that goal, what kind of problems are you still having, and how will that get better over time? Yeah. Good question. I think the kind of the main thing is, there's there's not a single way at the moment where we've agreed upon on in an enterprise context how agents should be interacting with each other. Obviously, there's different protocols out there now. We've got MCP and A2A. Both of these are, I would call, relatively immature protocols. There's a lot of security concerns, around them. And so that's probably the the biggest hurdle at the moment into getting things into production. Right? We we can connect to these agents, but it's like from an architectural standpoint, do we meet all of the specific or do these protocols meet all of the, specific criteria? And so I think as these protocols are maturing, we'll be able to see much more complex systems, and more centralized systems, over time. But I would say that that's probably the hardest part. Once we can connect them up with MCP, then, then it becomes relatively easy. But it's about how do you expose that, a to a MCP? How do you get that connected together? That tends to be the the hardest part. And then Vojtech said, does agent hub support agents external to Dataiku, or does the main orchestrator agent need to be a Dataiku one? Oh, that's a good question. So at the moment, I would say the main orchestrator needs to be a Dataiku one, and then it calls the other, external agents as tools. Yeah. That's that's how we have to interact with, with it at the moment. But, yeah, it's a good question. And then Steven asked, how are you incorporating DLP into the agent response, relying on the destination for security and DLP? Oh, and that's a that's a difficult one. I mean, I don't even know what the LPN is, so that goes beyond me. But that's a topic I'll research into. So, yeah. And then asked, how can we set feedback mechanisms if the output is not correct and we need to fix it? Yeah. Great question. So it's actually a topic I kinda skipped over. So within agent hub, we've got the ability to provide, thumb up, thumb down, and very soon we'll also be able to, essentially allow you to have, more like a, a custom response. Right? So you're essentially relying on feedback of experts that are using your agents on a day to day basis where they can give thumb up, thumb down, and then we can investigate that specific run or the multi turn conversation. And in the future, we'll be able to have much more granular feedback. And that's all displayed within the agent hub platform. So you can see you have an individual agent level, enterprise agent, or quick agent, how how they're being used, how often they're being used, which is a really good statistic as well, and then the overall feedback of, of the agents. I, saw a a really interesting question in the chat, which I might pull up here, which is from Mohammed. He said, is your platform used by business people or only technical engineers? And what, the Apple business people can get from it? And I really like that question. It's a question we get quite a bit. And it's called to our philosophy. So, again, I feel from the very start there's been a platform, that believes in democratizing, analytics, data science, and now, of course, agentic AI as well. I think it's always been important to be able to bring in diverse sets of skills, but also backgrounds, into, into a single project to be able to get the most results from from, from doing that work. I mean, there's there's gonna be many, many studies that have have proven this. So that's really the philosophy. It's like, hey. How can we bring in the subject matter experts who know everything about that specific area of the business together with these really smart technical people that know a lot about this technology, this very advanced technology, to get the business output that we need. Right? We need to essentially be driving ROI. Own. So the idea is bringing everybody together, allowing everybody to speak the same language, interacting within this user interface to, to drive those, those results. And that ethos has been from the start. Right? Before LLMs, before AI, when we were thinking about traditional machine learning models, that was really impactful because the business can come in and say, well, actually, that traditional machine learning model isn't working as it should. You actually got this wrong. The data wasn't clean, and was able to kind of help those technical people to get to that that better result. So I think Adjenic AI and LEMSA have only enhanced that. Right? Because the best people to write these prompts are not technical people. I mean, myself, I'm pretty good at writing prompts, but I have no idea what supply chain is. Right? I need a supply chain specialist to come and help me build that supply chain agent. I can't build that myself. So, the platform is for everyone. If you wanna code, you can code. If you wanna do stuff visually and be very efficient, you can do that as well. Okay. There are many questions. Okay. Victoria asked, how often do you update your ground truth questions and answers reevaluate the agent to LLM mapping as everything is changing all the time? Yeah. Really, really good question. The more ground truth data you can get, the the better. So, ideally, you wanna continuously be adding ground truth data over time to be able really to test the limits of, of your of your agent as well to see where where it performs really well, where it doesn't perform really well. You can do macro analysis. And this is something that I do a lot, right, where we run evaluations on, on our agentic systems and then combine all of those results, put that essentially together in in one single dataset and then fire that off as one cohesive dataset to, to an agent and say, hey. Can you tell me where are agents performing really well and where it's not performing well? That gives you, like, a really nice way where you can quickly at a macro level, you know, potentially have hundreds of ground truth q and a, see where the where the limits are. So that's a technique I've I've been using quite a bit. And then how often do you monitor is really up to the system and how critical the system is. There's obviously, a cost implication. If we wanna start doing evaluations over time, if we we've got a lot of ground truth, it can be quite cost intensive, to run that. But I'd say it's it's pretty critical to run it quite regularly, to be able to make sure that you both your agent is working, but also the systems that those agents are connected into. We often kind of, when we think about agent evaluation, we're just thinking about the output of the agent. But oftentimes, the systems that they're connected into can fail. And your your databases might might might fail because somebody's running a massive query. And then suddenly the agent might be able to do the job, but the underlying system isn't providing it with the information that it needs to be able to perform that job. So, yeah, as often as, as possible, really, would be my my recommendation at q and a, ground truth q and a, as as much as possible as well. Like, the more ground truth q and a you can get. And then Victoria had a follow-up question. Do agents learn from their own responses, or are any human edits made to the results? Yeah. This is, a bit of a philosophical topic. So not yet is the answer on do they learn from their own responses. There is this idea now going around of long term memory. Right? Agents learning from its own interactions with an organization. Maybe it would be super interesting for the agent to learn from my interaction with it so that when Julie asks the next question, it will be able to do that in a better way. But then, obviously, there's a lot of kind of implications on data privacy and, like, you know, potentially jailbreaking the agent to be able to get that history to understand where that memory is coming from. There's a lot of thought on it right now on how you can build these self improving agents. But right now, that's not not built in the platform potentially in the future. But, it's still very much being researched and best practices are being identified. I think ChatGPT does it very well, but that's on an individual level, where it can reference past chat quite well. But it can also sometimes be annoying. So, yeah, it's, it's it's something that I think will come as of improving agents, but that's kind of at the frontier at the moment. And then Steven asked, LLMs are covered, but what about visual language models and audio language models? Yeah. Really good question. So VLMs baked in natively into the platform. So specifically, we use that quite a bit when we're doing, text extraction. So if you've got a lot of PDFs that you want to start vectorizing, VLM, so Visual Language Models can be really good at identifying and describing PDFs that have a lot of tables. That was traditionally the limitation with OCR where it wouldn't be able to interpret the context from tables. It would just extract text. Right? So if you had any graphs or anything like that or images, then your vector database would pretty much fill and become useless. And so within Dataiku, the embed documents recipe is super powerful. You can essentially link it to various different documents in SharePoint or s three, wherever that might be, and then choose a strategy that you want to apply, whether that be OCR, VLM, or a combination of the both based on certain conditions. You may wanna say, hey. For some documents, I wanna do VLAN. For some documents, you wanna do OCR. Reason being VLAN can take quite a while, and it can be, token intensive as well. So, yeah, that's baked in the platform. Audio isn't, yet. We've not seen a huge amount of, let's say, enterprise use cases on on audio. That's, I think, for now, mostly on the consumer market. The the use cases that I have seen that have leveraged audio, like call center, call analysis, this type of thing, we would use traditional Python to to be able to use that. And then, to circle back to ground truth, Tom had a question. How do you integrate the ground truth testing? Is this something that a human has to do, or is this automated? Unfortunately, it's still very much what a human has to do, but maybe fortunately. Right? I think that ground truth, the the whole the whole idea is that, the whole idea is that it comes from a human, an an expert. How would an expert answer that? But it it it is sometimes interesting. And there's been a couple of examples where we've had ground truth, and I've worked on use cases where we've had ground truth from, you know, 10 analysts, let's say, that they're performing certain tasks. And when we did a macro analysis, we actually said, well, the NLM is not performing great. And it was a very specific use case, because they're not answering the same way that the human, was answering. Okay. But then interestingly enough, when we brought the result of the LLM to one of the leaders, in in that organization, they were actually like, hey. This is actually better than what I was receiving from the humans because the LLM had a different way in which it thought about the problem. I can't go into too many details here, but it was really interesting because it highlighted how humans reason very differently than than the LLMs. And, in in this case, it was, it was actually looking at, like, errors, and it was able to to identify that instead of sometimes blaming like a human, it would blame an underlying system. Right? If a if a human was able to make a mistake, was it the human's fault or was it the system that they were using? And so that was super interesting. We were able to kind of course correct with prompt. Right? So we can say blame the human. If an if a if it if the human makes a mistake, then that is the root cause of of that issue. But, actually, the leadership was like, we should keep it as it is because we wanna blame the the underlying system. Because if a human can make mistakes, that's an issue in the underlying system. So, yeah, it's, it's an interesting one. Ground truth, we we've gotta I think it's it's gotta stay human. Otherwise, you kinda lose the definition of ground truth, and it's collected quite manually, at the moment. And then maybe we can take these last two questions, to wrap up. But if you happen to know what percentage of your agents have you fired so far. Yeah. Interesting. Not it was actually zero. Probably in in in jest, but, I think that, you know, agents. aren't perfect, and, you know, I think that's the nature of these nondeterministic, models. You know, you can ask two questions twice and get two different answers, and that's alright. I think it's, you will see mistakes, and and I think that's why having a human in the loop to be able to identify when it's going well, when it's not going well. Of course, correct agents in multi turn conversations is is key. But also highlighting when it isn't doing well and engineering the prompt to fit that. I think prompt engineering, especially when speaking about agents, is is absolutely critical. That's personally why I like multi multi agent systems better because you can get really specific on those individual agents and really specify the prompt versus having one agent with a massive, massive prompt. We've seen various examples where the bigger the context gets, the bigger the prompt gets, essentially. The worse the agent is, performing, and also the more tools an agent has, the worse it is at performing. So we haven't fired any yet. They're very powerful tools. But, again, they are they're not magic. They they are tools. So we need to use them in in the right way. And then the last question, what is the vision for this product, and what new features are on the road map? Wow. That's a massive massive question. I'm probably not the right one to, to be able to to to share that information. A lot of exciting stuff. That that's what I can say. I've been at Dataiku over six years now. I never cease to be amazed at what we're able to release and how quickly we're able to do that, but do it in a way where it actually works within enterprise, organizations. I love the fact that we're tech agnostic. I we're all one of the only players in the market that enable you to do this at scale and choose the technology that you wanna leverage, not be locked in to to any given technology. And especially in this landscape, I think it's absolutely critical. You can't put, put your bets on on one horse at the moment. It's a it's very much a multi horse race of who's gonna get to, to essentially AGI, first. So you wanna have tech optionality baked into whichever platform, you you choose to use. But a lot of exciting stuff. It's part of what my team is involved in. We're constantly getting feedback from customers that are really deploying agentic use cases that aren't just looking at it from a POC perspective, but, like, hey. All this stuff that we're seeing online is, like, really cool, but it's certainly not production, ready. And so really honing in on how do we actually start getting ROI from these agents. A lot of what I think the the product direction in will be around that over just doing these moonshot POCs, which is personally something that, I think is is is the right direction. Awesome. Thanks, Chris. And I know that, like, in the next version of, DataIQ, there will be a lot more agent evaluation features, and we're working on, like, agent control and more agent orchestration. So definitely please all tune in. And if you're interested in learning more, please feel free to contact us, reach out, check out our blog, and, we'll be having a lot more future webinars on agentic use cases, and different features in Dataiku and demos. So please, do come for that. Cool. Thanks very much, Eran. File.