Video: Live Demo: Turn Agent Experiments Into Enterprise ROI with Dataiku | Duration: 3100s | Summary: Live Demo: Turn Agent Experiments Into Enterprise ROI with Dataiku | Chapters: Welcome and Introduction (7.12s), AgentHub Key Features (133.005s), AgentHub Platform Benefits (291.875s), Agent Demonstration Overview (401.875s), Metadata and Tracing (1854.635s), Exploring Trace Explorer (1931.36s), Sharing MCP Tools (2027.135s), Versioning and Deployment (2173.3298s), Data Input Capabilities (2298.2952s), Sensitive Document Safeguards (2355.855s), Authorization Integration Explained (2439.27s), Access and Feedback (2512.5251s), Agent Evaluation Features (2695.045s), Data Accessibility Rules (2802.405s), Concluding Agent Hub (2898.1401s)
Transcript for "Live Demo: Turn Agent Experiments Into Enterprise ROI with Dataiku":
Hi, everyone. Welcome to our live demo of agent hub. We're super excited to have you all here. Before passing on to our speakers, I just wanna give some, introduction and housekeeping, information. For reference, we will be sending the webinar replay as well as, the deck after the fact. So, feel free to ask questions on the chat during the webinar. Our speakers will be able to answer. And, again, we're super excited to have you here for the first agent hub live demo on how to unlock business ROI with agent hub from Dataiku. So onto our speakers, and, enjoy the webinar. Hi, everyone. I'm Julia. I'm a product marketing manager at Dataiku. Oh, and I'm a product manager at Dataiku. Just changed that title. So just to level set, our GenAI and AgenTic capabilities are part of Dataiku's universal AI platform. That means that you're able to connect your agents to your data analytics and machine learning when appropriate, and it also means that your agents are built on top of the LLM mesh. Agents are only as valuable when they can integrate with your data analytics and other AI models, so they need to be governed and controlled in the same way. And we're gonna talk about how agent hub helps you do that today. So in a sentence, agent hub is a collaborative workspace where every employee can find, use, and create approved AI agents, while IT retains full control over access to models, data, tools, and the agent life cycle. Agent Hub can be broken down into four core capabilities centered around the ideas of controlling experimentation and scaling innovation. The first is that agent hub is a one stop shop for enterprise grade agents. With agent library, every employee can access approved trusted agents all in a single place. Access rights are enforced by IT so employees can get started faster without putting data, compliance, or budgets at risk. The next is self-service agent creation for any team. So building shouldn't be limited to just technical experts. In agent hub, business teams can create a quick agent to build what they need and when they need it with no code necessary. So business users can create an agent in seconds using natural language on a single page. They can build with one of the 20 plus prompt templates for common business use cases and leverage approved LLMs, data, tools, and agents already governed by IT. Instead of waiting weeks in backlogs, business teams can solve their own problems while IT keeps control of guardrails, costs, and approvals. Each Qlik agent automatically becomes a Dataiku managed asset, allowing IT to support troubleshooting and share them to the rest of the org. Next is agent hub's multi agent orchestration. So the orchestration layer here turns agents from stand alone utilities into a dynamic workforce. Simply ask any question in the user friendly chat interface, and it will automatically get routed to the right agent or coordinated across multiple agents for complex tasks. Measurable ROI is our last one, and agent hub provides a really clear path from early experimentation to measurable business outcomes by helping IT teams centralize agent experiments, monitor and provide visibility into all agent activity, make informed decisions, and manage shared permission sharing permissions about which agents to further scale enterprise wide and which to retire. With all of that said, AgentHub brings together all agent experimentation, distribution, and governance into a single place. For business teams, that is the freedom to create and use agents tailored to their needs. And for IT team, that is full oversight of access, data, models, and life cycle management. A huge benefit of agent hub is that it's part of the broader universal AI platform. So flexibility, governance, and other features with Dataiku are extended to it. One example is Dataiku stories. So once you have an agent answer a question or execute a task, the work doesn't end there. You can automatically create dynamic presentations directly from the agent's outputs using Dataiku stories. Many of our customers are benefiting from agents already. Recently, Solidigm faced a high volume of IT support requests with over 2,000 users across 33 teams submitting more than 2,000 ServiceNow IT tickets per month on a wide range of topics. They had different teams for escalations and topics covering literally anything IT related. Managing this was super was super time consuming at scale and created a lot of bottlenecks. So to streamline resolution, the data team built Solidigm GPT, a support agent using Dataiku that integrates Snowflake Cortex, internal knowledge articles, and VisualML to automate the ticket triaging and escalation processes. And so far, the team has had over a thousand Solidime users log in and actually submit support requests to the agent. Now that we've gone over that overview, I'm going to hand it over to Guillaume to go into a deep dive demo of AgentHub. Cheers. Thanks, Julia. So just to give you an overview of what I'm gonna talk about today, I'll go through different, customer stories and start from being very simple, starting from a business user, that is gonna be a customer success manager, and I'll end with a data scientist. So the goal is kind of to give you different flavors of what you can do in different industries and with different, personas. Okay. Let's actually start now. I just can't share my screen. Alright. Thank you very much. Alright. You you should see my screen. Can you confirm this? Yeah. Alright. Perfect. Now right here, we're in the interface of, agent hub. We'll start with the first, story. Let's imagine I'm a customer success, manager working at an IT company, for instance. This is the interface where I can find all of my agents. So this is the agent library. I can find both the enterprise agents that have been built by, designers and also some agents that I can be myself, but we'll come back to that, later. Now for the month, what I can see is that I actually have one agent that can be used to search, customer data and another one to predict customer churn. Now I can imagine that this one is using a model to predict whether the customer, is going to churn. Now for the sake of this demo, I have a first question, and I've received an alert about a client named Singleton Williams. I want to get some informates and also, well, predict the probability that it turns. So what you can see here is that I've just selected two agents in the chat. So what's happening behind is that I have an orchestrating, LLM that is choosing the right agents, to answer and the sequence of action. Technically speaking, what's happening behind is that I'm using the long graph library, and I'm calling the agents as tool using the React framework. Okay. So I can see that okay. So I've got the correct, information. What's interesting is that right here, I can see the sources. So this is the exact, source, from the dataset that it fetched. I don't really know what's happening behind, but here it is. And here are the traces. So I could see that it called the search customer data agents and the predict customer churn. They also used a, a tool. Alright. I see that it didn't actually give me the probability to, that it churned. So, yes, it can give me the like, yes, that this customer churns. Alright. Perfect. So it's calling again the, orchestrating model. Now the first value, of this is that, normally, to actually get this data, I would probably, send a message to, data scientists just so that they do an SQL query. Here, I get it, in seconds. The second value proposition is that, I can directly call, a model, a machine learning model, but without really knowing what's happening technically, behind it. Right? Alright. So I see that Singleton William is predicted to be at a risk of churning. Right here, it details why. So the average sat satisfaction score is quite low. Revenue is moderate. Featured adoption is average. Now this is great information, but there's one thing I might want to do, which is actually write, an outreach, email to this customer to make sure that, it doesn't churn. And I want this email to be, personalized to my company. Now either I could ask my admin to do it. I could also do it manually. I mean, I do this every day. That's my job. But this time, let's actually, create an agent, to do this. I'll call it outreach email agents. Now I prepared the prompt because I don't want to write it out, but you could, use the prompt library to, take some, prompt that are already written for you or just to structure, your prompt. I'll use this one. On top of that, I have an LN that has already been selected. Here, my admin only selected one for me. I also want to upload the documents. And more precisely, what I'm uploading right here is a value proposition document. So what does that do? This value proposition document, has been crafted by my sales team to explain our value proposition for various industries. So for instance, for, the industry, for retail, well, all type of industries. So what you can see is that right here, I uploaded it. And when I click on process, documents, what's happening in behind the scene is that it's actually, chunking it, embedding it, and so creating a knowledge bank that I can use. As a business user, I don't need to know what's happening. This is being done automatically, and as you can see, it's already embedded. So it's relatively fast. On top of that, I don't want it to just write an email. I also want it, to be sent. So I'm gonna select the, email tool that has been designed by my, my administrator. And this agent, overview is just, the description that I will show, in the interface. Here, I don't want to write it. It's a bit time consuming. So I'm just gonna authorize it using an LLM. Alright. Perfect. I can see that I can actually test it out. Let's try it. What you can also see is that, this agent is versioned. Here it's saved. It's in draft mode. I haven't published it. And it means that, if I change it, I can actually go back to a previous version. Alright. Let's see this. This is the test interface. So this is loading. I can see it's using the email tool, and it has crafted an email. Okay. I actually just got it, so it is indeed working. Now let's publish it. Okay. Now I've actually already built this, agent right here, so let's use it, in the conversation. I'm choosing my outreach email agent, and I'm gonna use this prompt. So it looks like there is a high churn probability, with this customer. I'm just asking you to write an outreach email. On top of that, this outreach email will actually use, the value proposition framework, I gave it. So right here, it's calling the agent, the outreach email agent. And what's interesting is that you can see that in my chat, I have three agents that are selected, but only one is called. So this is thanks to the head LLM with selecting, the agent it's supposed to use. Okay. So I just received the email. So, indeed, it's for the singleton Williams team. You can see that the email is indeed tailored to the customer, and the industry and that I haven't personalized yet the the template, because this is an anonymous company. But, that's okay. So what we saw right here was, how a business user could use the, interface. Let's look at what's sitting behind it. Now behind it in the flow right here, what I have is two agents, my churn prediction agent and my customer data agent. They are using a machine learning model that has been trained using these datasets, rather small dataset, by the way. And the other one is looking up, information in this other dataset. Now the way these agents are built is using visual agents. You can see that the interface is really simple. I just need to choose an LLM, put the instruction in, and add some tools. These tools, similarly, what I've used here are visual tools. If I go to this tool, for instance, this directly connects to the model. I just need to put a description and save the model. So that's one of the proposition of our platform, giving you ready to use an easy to configure, tools. Now in some cases, this work this works because this is, these are simple, datasets. So the only tools I'm using is a a dataset hookup tool. In another use case that we look at, the datasets may be actually SQL datasets with much more rows. For for instance, right here, we have, 10,000 rows and, 17 columns. So let's move on to the second use case. I'm just gonna go back to my, agent library, and this time, I'm gonna use the financial analyst agent. So these agents, can do SQL queries, to this data set, which is rather complex. But more than just fetching some data, it can also do some aggregation. Alright. So I'm gonna use a quick start to do that. What's interesting with these, quick start questions is that what we see is that, users always start by asking quest asking the agents what can you actually do. So we decided to give these agents directly to the users. Now in this instance, as a financial analyst, I want to add the agent to break down the asset classes in each of my region, by, NVV. Alright. Perfect. So I have my net book value. As you can see, I've got my result right here. But on top of that, it's not just fetching the, the data. It's also generating a chart, a chart I can interact with, a chart I can download, as a PNG, but I could also download it, as a CSV, which I could, afterwards modify, in Excel. On top of that, I can also, open up, create some insights, sorry, in stories. So stories is our, equivalent of, PowerPoint. And what you can see that when I do that, what it does is that it has access to this finance, datasets. So I gave access, to this dataset in the first place. Everything is set up correctly, and it's creating a presentation. Now let's kind of talk about, how this would work, in real company. Imagine, this financial analyst, I might want to first, have a feeling of what the data looks like. Right? So I would ask first a few questions, maybe have this visualization. I can also confirm that this is the correct data looking at the, the sources or potentially download it to do some modification, in Excel. After that, I may I want to actually, build some slide, for some decision making, discussions. What's interesting here is that this directly embeds, some graphics, graphics that are directly connected to your data, data that is, real time. On top of that, it's not just data. It's also some textual insights to explain, what you are, seeing. Just to finish on what we have, in agent hub, I could also give some feedback on whether this is useful. Now for instance, I don't know what you saw, but right here, I don't have the units. I only have million. And, well, I mean, that's a bit of an issue. So I want to let, my admin know that agent didn't show the units. So it means that direct in the interface, we have a feedback loop that is, integrated. More than that, we also have monitoring, dashboard. So this is the view that I would have, as an admin. What you can see right here is that I can see all the enterprise agents, in my instance. All the user agents that have been created but also shared, we go back to that. And so those that are still in draft mode and that have not been saved. I can see how much these agents are used, looking at different time steps, so daily, weekly, monthly. What you can see here is that I've been using, these agents a lot, right before the demo to make sure they actually work. So, you can actually see that, and you can see, whether, the feedback loop. I can also see, some detail activity. So for instance, I can see that the the most used agent is the search customer data agent with an average feedback of, o point eight, which is not that bad, and that the outreach, email has a a great, feedback. The idea behind that is that you can actually see which agents are really used, which agents you may want to work on, improve, or share to a wider audience. Take the outreach email. This maybe is a great agent, so I may want to share it with, other users. So I'll share it with Julie, for instance. She'll be able to, use it directly. But, usually, since in my team, maybe my admin wants to share it to the whole company. So what they'll have to do is, actually, build a proper agent, in the flow and then declare it as an enterprise agent to share it to, a wider group. What I've shown so far are rather straightforward, agents. I want to talk to you about agents that sit in between different ecosystem. Now let's go to the last use case. This time, I'm a data scientist working, in supply chain. And perhaps I don't want to actually, consume my agents in a conversation, so like in agent hub, but I actually want to use it in batch. What I have right here is a list of, suppliers. And for each of these, I want to have a tool that searches the web to find different news. Let's say about geopolitics, quality, issues, or local strike, and I want an agent that automatically assesses the risk for its suppliers. If this, if the risk, is important, I wanted to send a message to my, purchasing and supply chain, team, so that they're warned and they can actually, you know, contact the supplier, potentially, change it to mitigate, the risk. Now this time, I've used another type of tool. Instead of using, classic, tool, I've used an MCP tool. So what you can see is that in that Dataiku, I can very easily, build, MC MCP tools. So right here, I just put my, API keys and some basic, arguments to actually fetch the, MCP server. If I click on this, I'm gonna get the whole list, of tools. And it's also super easy to equip my agents with these tools. So if I take this visual agent, I can see that right here, I have my MCP tool. I have my, tool that sends the message and a tool that also logs, these reports in a dataset because it's quite important to have a trace of this, history. Now so far, what I've shown is are only purely agentic, agents. And by that, I mean that the only way to actually control them is through a prompt. By the prompt, I could tell them to use this tool, then this one, then this third one, but I don't have a strong control, a strong oversight on what is actually used. So this time, the way I actually decided to implement it in code. So this is my collagen. I'm using, long graph, and what's interesting is that I can define some precise steps. So in nodes, here, I'll have four nodes, and the agent will execute them, exactly in the order that I have defined. The way I call him, the agent, is through this prompt recipe that is connected to my dataset. Here in this prompt recipe, I only define some prompt, inputs. So the description and the columns. Right. Let's run it, and let's look at the result. So I can actually see that the job right here, is running. Just gonna take, a few second. Hang on. The result will then be added to this, dataset. Okay. Let's actually look at it. Okay. Normally, it should be good. Alright. Perfect. So as you can see, I've got all my results for, all of my suppliers. Here, I'm not overwriting. So I'm appending new roles, every time to make sure I actually, get a trace. Let's look at the LG Chem for instance. You can see that I found a medium risk, And the reason is that we're navigating some financial, adjustments. So I'm not making this up on this, actually, fetch some data, on Nynum. And this could be, a risk. Now if I actually have a high risk, this sends me a Slack message directly. So you can see I just received this. And for instance, I actually got a message for Foxconn, given they are, having harsh conditions for their, employees. One thing I also want to, underline here is the importance of traceability of your, agent actions. I'll just go back to, agent hub. So when it comes to traceability, we have these traces, but these are summarized traces. You may actually want to open up the traces and have a full log of the trace. So this is what we have in Trace Explorer. I know exactly which tools were called, how much time it took, and how man how much money, that cost me. And lastly, what I've been showing so far are rather straightforward, agents, agents that have one or two tools that are directly connected to a dataset. But it's also possible to create a bit more elaborate agents. So let's take this one for instance. This is a content writing assistant that has three sub agents and a set of different KBs and that can directly generate some marketing content. This is also purely agentic. One of the biggest value proposition of Dataiku is that I can also call my agents in a deterministic fashion. So for instance, here, I'm preparing some data. I'm calling these agents right here and right here in a very precise, manner. So, overall, what I wanted to underline is that in Dataiku, you have different ways to, build agents and distribute them. You can build them using visual, items, but also code. You can distribute them in ready to use interfaces, like agent hub, but you can also build custom, web apps. And lastly, we have, stronger, control and governance features, which are really important in an enterprise setting. I'll stop sharing my screen, and I'm ready to take some of your questions. Awesome. Thanks, Gilyam. So some questions that we've gotten already. Mark wanted to ask, just to clarify, is it possible for these agents to execute scenarios within Dataiku to help answer questions? Is that how the customer churn worked? So, customer churn didn't use a scenario. The way it worked is that it directly called, a model, and, it called the model, and used the data that we pulled, to predict whether the customer was likely to churn. Going back to the first question, yes. It's possible that agents, execute scenarios, And more and more, we're working to make sure that, agents can actually use tools, inside the Dataiku. And, actually, conversely, it's also possible to use scenarios, to call agents. And for me, this is one of the best, ways to use them, when you use them, in batch mode. So for instance, if you take the last, use case, this, supplier risk monitoring agents would be called, I don't know, every month or every week using a scenario. Nikolai asked what kind of model is used for churn, LLM model or traditional ML model? No. For churn, we're using a traditional ML model, and I think this is also one of the strong differentiator of Dataiku, having in the same place the data preparation, the machine learning part, and the gen a part part. We're convinced that, machine learning still has a stronger place, to play. And, obviously, Gen ai wants, replace a traditional, ML model. Peter asked for the financial agent, is the underlying data described in a data catalog? So are you asking whether there's some metadata? So in this specific instance, I don't think so. Wait. Actually, let me share my screen again. Let's have a look. I like precise answers. We might have a precise answer. Right? Oh, oops. Wait. Sorry about that. I don't know what to do. I'm just gonna go back. That's a bit annoying. Sorry. Okay. No. You can see that there is no meta metadata, but we actually have a module right here, to generate metadata. And I would agree that if you have a an SQL tool, the best way to have a great result is to have good metadata. And for me, the best practice is, working hand in hand with AI to first generate metadata and then review it manually to make sure that it actually, is close to reality. Mark asked, could you cover the trace explorer in more detail? Agents' calls can become quite complex with a user facing one. The orchestrator various task agents would be nice to see that again. I know. Agent calls can become quite complex. Yeah. Of course. So let me share my screen again. So first okay. I'm just gonna go back to it. Okay. Trade explorer is right here. Okay. Perfect. So there there are different ways to, explore it. What you can see is that you have different items, that are tied to, a code object. So right here, you'd have in pink, the agent in green the tool. Here you have all the codes, but I can also remove them. And then you also have the, events. The way this is, constructed is that in the long it's it's using events in the long graph library. The idea behind that is that you can see exactly the prompt that was, passed on to, different tools. Okay. Let's see. We do actually we do have one. Right. Okay. Perfect. So you can see exactly what was passed on to the, LLM. So right here, you can have the, user query, the different, arguments, and so on and so forth. And also, right here, you'd have the breakdown on how much time each steps, took. The way these two would be used, definitely, it wouldn't be used by a business user. This would rather be used by, let's say, a a full designer. It's a great way to debug. Sometimes when you build agents, they're stuck in a loop. You don't know why. And so the best way is to actually look at the the query. Yeah. I can't say much more about Trace Explorer, than than that. Clement asked, can one person build MCP tools and then agents built by another person can discover them and use them? So, yes, one person can build, MCB tools. The interesting part is that once you've built, an MCB tool, anyone can use it. It's, it's visual. I mean, I can show you how how this will work. It's really straightforward. It takes, thirty seconds. So let's take a basic agent. I'll call this MCP app scraper. I'll select an, LNM. Okay. Let's take close for instance, and I'll choose my tool. So this is my whole list of tool in my project. And what you can see right here is that I have my list of, MCB tools. For instance, this time, I'm just gonna take my, brief search tool. Right now, this is saved, and, I mean, it's already working. I I mean, I would need to put a a proper prompt, but anyone could actually, do that. Now a small caveat is it's important to differentiate the personas. Imagine I'm in, agent hub. Right here, as a user, when I create an agent, I don't have access, hub right here. I don't have access to the, MCP tool that is, in the same project. And that's actually a good thing because, this, MCP tool is running web queries, calling an API, and this can be a cost for the, the company. Right? This is also reaching some external data. So I mean, maybe I don't want to, allow this. Let's actually look at how this is done, in agent hub. So in my edit section right here, I can declare the tools I want my users to be able to use. So for instance, if I wanted them to be able to use these MCP tools, I would just have to select them like this. Save. And now anyone could use it. So let's, let's show that. Up. Voila. Now anyone could use that. Awesome. Ezekiel wanted to know, is it first question, is it possible to automate the deployment through a pipeline? And second question, is there a versioning system to manage the evolution of our systems? Hello. Let's start with the, versioning, and then we talk about deployment. There are two subjects. If you take classic, enterprise agents, this is a a an item built in a project, in that Dataiku. Yes. There is classic, versioning with a design node and auto an automation node. And on top of that, we also have a a govern node. So the classic life cycle would be to test, design, and iterate on the design node, then govern the project and push it to, automation node. And on the automation node, I would add it to, AgentHub. When it comes to quick agents that can be directly built by, business users, the the versioning and the testing is directly done, in agent hub. As you can see, it's be because these agents are rather basic. They can access, data, and the tool that has been selected by the admin, but you don't need, a CICD process that is more elaborate, than that. The first question was on deployment, Julia? Yes. Is it possible to automate the deployment through a pipeline? So when it comes to deployment, the way it works is that, you can bundle your different projects and then directly push them to, the, automation loan and declare them, in agent hub. The idea is that also you will have a single entry point, so this would be agent hub. And the way you would set up, the security permission is by defining user groups, and giving these, user group different access to your different agents. And If you don't have access to an agent, you don't see it at all, in the library. Okay. Great. And then another question someone has is Christian asked what kind of data can be used as input and how much? I see a Google Sheet, so it can read Excel in Google Sheets and interpret them as well. What kind of data can be used as inputs and how many? Well, generally speaking, we can integrate with any type of data. The real question is, what do you actually want to do with this data? Right? If your data is in history, yes, we can integrate with it. So it kind of depends on what you want to do with it. When it comes to direct in chat, a conversation, for instance, generating graphics, The only way to generate graphics is by using, an SQL tool. We have, further things that are coming up, on the road map, but generally speaking, there is no true limitation. And then Michael asked, is there any guardrail to prevent the upload of sensitive documents in case the user creates an agent with a third party LLF? Yeah. So various thing various things on this subject. Let's start about guardrails, generally speaking. Generally speaking, guardrails, for to do a PII detection, prompt injection, or, I mean, all of these things, these are done, upstream. We have, a module called guard services, to actually, do that and, I mean, do, way more things. When it comes to uploading sensitive documents directly in agent hub when you create, agents, it's possible to, do two things. It's possible to disable agent creation by, your users. Maybe you only want your users to consume agents that have been built by, proper designers. If you actually want them to be able to create some agents, you can only select an on premise, LLM, for the, these agents that they can build. So the the all of your data would, you'd would stay, I mean, wouldn't go to the cloud. And then Wayne similarly asked, in terms of authorization of who has access to which tools, can we integrate this with our enterprise identity system based on Entra with Okta, for example? I don't I won't be able to give you a very precise, answer, Wiena, but let me try to give you one. So the way permissions work is that, currently, the application runs with the identity, of the back end. The, user, ticket information is passed on to the agents, which means that, you can use these in any tools to restrict the user of tools. Usually, the way we integrate with other, security based systems is by mapping existing groups in other systems, to, Dataiku DSS groups. I hope this is clear. Nikita asked who has access to prompt text except for the user who authored them and the outputs. So what happens when you create, this quick agent is that it creates a project, in Dataiku DSS on the instance. This means that, the admin can have full control and oversight, on, what's in this agent. When this agent is shared to other users, they have no information on the documents that are in these agents nor on the prompts. So it means that the only people that can actually see it is the person that has created it and the admin. When it comes to the I'm sorry. When it comes to the outputs, so the the proper logs, the only person that so you're the only person that can access your conversation history and the admin. All the logs are stored in a a database. And then Essen asked, for an enterprise with hundreds of users, is there any limit on the number of agents that can be created by users? Currently, that's not the case, but, it's possible that we added, to the road map if that's a requirement. And then, Veronu just asked, can you elaborate on human in the loop and whether feedbacks are used to improve the models? Hello. So two two very interesting topics that are not necessarily easy. Currently, the way human in the loop, is implemented is, through, agent prompting. So you can ask your agent, ask me confirmation before sending the document, ask this validation before doing that. I think that's not entirely satisfying. So this is on our road road map, for the upcoming months. We're trying to have some, buttons to actually validate, the actions of agents. When it comes to how our feedbacks, used, it's used in different ways. So first off, all the logs of the conversation can be used to evaluate the agents, make sure they're not drifting. The second way feedbacks, can be used is directly by the admin to improve the agents rather than the model. This could mean that they may want to change the, underlying model. This could mean that they could change the structure, of the agents, or they could also altogether change the way you, interact with them. Yeah. This is very broad. Usually speaking, it's easier to, answer this question, when working on a specific use case. Okay. And then someone else asked, going back to the Trace Explorer, Derek asked, can we also tie traces to cost per outcome and model drift over time so we can spot when an agent gets pricier or less accurate? I think this is done on the agent evaluation side. I don't want to say, I'm not 100% sure of the answer, to be honest. Yeah. I know that, in terms of product road map, from a Dataiku perspective, we will be working on more agent evaluation, orchestration, and control, features in the coming months, so you can definitely keep an eye out for those. Let's see what other questions we have here. And this is a human in the loop again. So do you support human in the loop, a user determining an action in a multistep process in Dataiku? Well, currently, the way it could be implemented, by using co agents. So, yes, that currently work. But if you want to implement it, using visual agents, that's not currently supported. Or as I mentioned, yes. You can do it, by modifying your prompt, but it's not 100% deterministic. Igor asked, can how do the agents observe data accessibility rules and permissions? Is that configured elsewhere, in a DG tool, for example? Wait. Could could you repeat the question? I couldn't find it. Yes. He said, how can the agents observe data accessibility rules and permissions? Is that configured elsewhere, in the tool or in a DG tool or in Dataiku? Okay. That has access accessibility rules. So two things. Just to reiterate, when it comes to permissions, you can only see the agents that are in projects you have real access to. So as a user, I can't see the agents I'm not supposed to use. So I just don't see them, so I can't use them either. When it comes to data accessibility, so imagine I have AKB with different types of documents. Some of them, I can use them. Others, I can't. This is done, directly at the knowledge bank, level. This is also possible with, other systems like, SharePoint. We're actually releasing a tutorial, this week on the subject. Awesome. And then James said when is agent hub expected to be released and under which version? So agent hub is expected to be released in the next version. That is, for Dataiku14.two, and we're anticipating that that will be available next week. And let's see what other questions we might have missed. Will there be any workshops in the future for more detailed use of these agents in creating them? Yeah. Definitely. And and I know this is gonna be, organizing, so already the case. And by the way, if you, are already, clients and trying to, implement, agents, I'm happy, to support on the rollout of, agent hub. And feel free to also reach out if, you got some feature request, further feedback, things you you want to see on the road map. Let me see if there's any last questions that we missed. And if not, we can just wrap here. Bruno said, is it possible to orchestrate different existing agents to perform advanced reasoning? What do you call it? Advanced. Okay. Is it possible to orchestrate differently? Yeah. Definitely. So this is actually what, agent hub does. It uses the Langraph library and calls the agents as react agents. And the idea behind that is that when you call agents as a tool, you can call them, iteratively, so multiple times. You can also use the outputs of an agent as the input of another agent. So, definitely, today, we can, do some, pretty advanced reasoning. Awesome. I think that we've gone through all the questions. If anyone has any last questions, feel free to send them to a chat. Otherwise, we'll wrap here and give everyone eight minutes back. But if you're interested in going further with agents, feel free to scan either of these two QR QR codes. One goes to all of our existing webinars on agents currently, and then the other one goes to a quick three minute demo of Dataiku. We will be having an agent orchestration webinar coming in the next month, and a number of other agent workshops and content. So please keep an eye out for those, And thank you all for joining us. Thanks, everyone. Have a great day.