Name: Building an AI Agent in Dataiku
Uploaded: 2025-03-10T16:30:43.951Z
Duration: 52 min
Description: Building an AI Agent in Dataiku

Transcript for "Building an AI Agent in Dataiku": Hey, everybody. Thanks for joining us today. Really excited about this webinar. I do wanna make sure I give everybody enough time to jump in before we get started. So we'll give it about another minute before we kick off. Looks like we've got people from, all over the world calling in, so say hello to your fellow webinar goers in the chat there. Okay. I think we're about one minute after. We can go ahead and kick off. And, others who join late, just so you all know, the webinar will be recorded and sent out later so you can review it. So thanks for joining us today. Excited to talk about agents in Dataiku. I'm Christian Capdeville, so I lead product marketing, here at Dataiku. And, you know, really excited for today's webinar showing how to build an AI agent with Dataiku. So with me is Chris Burritt, strategic sales engineer at Dataiku. You know, Chris works with data teams and financial services organizations and has been at Dataiku for five and a half years. So he's an expert at navigating complex data challenges and guiding some of our newest customers, and maximizing the value they get from Dataiku. So today, you'll learn how to design and implement an AI agent Dataiku. We'll touch on integration with tools and datasets. I will share a few strategies for testing, monitoring, and iteratively improving the agent's performance. But before we get started, a couple of housekeeping items. So number one, the webinar is recorded. So you'll receive an email after the webinar with the recording and some additional content. And, two, if you have questions, go ahead and add them to the q and a tab, on the right of your screen, and we'll respond to those at the end of the session. So with that said, here's a brief look at what we'll go over today. Number one, we'll clarify what we mean when we say AI agent. A lot of definitions out there. Two, we'll give an overview of the thing we're about to build. And then three, we'll get straight into the demo to show you how you can build an AI agent in Dataiku. And in the end we'll have some time to answer questions. So what do we mean when we say AI agent? So there's a lot of definitions that exist, but really at the core we say an agent is an LLM powered system designed to achieve objectives across multiple steps and, with the concept that goes beyond just answering a question. Oftentimes, the best way to understand something, though, is to compare it to something else that we might understand really well. So let's talk about the idea of a chatbot versus an agent. Like, when you see when you use a chatbot, you ask a question, the chatbot gives you an answer, very simple. But agents do a little bit more than just answer the question. Right? They still may do the same initial steps as the chatbot in some cases. Let's say you're still using a rag pipeline to answer a question, but they're also able to go beyond this. So they can execute multiple steps, they can take certain actions. So in our comparison example here, we have chatbot on the left that can answer your questions about your company's expense policy, for instance. But an AI agent in this space would be able to do things like review receipts, check them against policies, flag any issues, route items for approval, make updates to systems, and even send follow-up messages if if needed. So the potential for agents to autonomously handle meaningful chunks of complex workflows really opens up a whole world of possibilities. We can, of course, automate existing processes, but there's also opportunities to completely reimagine business processes in light of what agents could do. So with that in mind, we're excited to show you exactly how you can get started, with building agents in Dataiku specifically. But before we jump into the demo, I think it would be really interesting for everyone here to get a sense of where this group stands with regards to adoption, both for GenAI chatbots and and for AI agents. So a couple quick polls if you wouldn't mind participating, and I'll I'll open them up. But first off, we'd love to know, do you have GenAI applications deployed to production today? And so this would be like, a rag question and answer chatbot. Maybe you're gonna ask questions of your company's documentation. And and deployed means, like, employees in your company today can go and talk to that chatbot, for instance. So we'd love to hear where people are at with this today. And I'll, share the results in a second here. And I will close that so we can go to let me see here. Oh, I think maybe I'll stop sharing so I can share the results of this poll. That's how it'll work. Okay. So we've got, I'll go ahead and close this. So we've got an interesting mix here. We've got 42% of people that have a GenAI application of some kind deployed today, 29, so 30% working on it, and, 28% not not quite there yet. Even split, pretty interesting. So would love to ask the next question. Let me stop sharing this and I'm going to ask the next question which is essentially, what about agents? So this is things beyond question and answer bots, but they can actually take steps. They can do autonomous actions perhaps, or beyond just answering the question, they can they can execute some step of meaningful work. Is this deployed in your organization today? I'll give everybody a minute to weigh in there. Awesome. Okay. Thanks for thanks for weighing in. We can see the split is a little bit different. This is the next horizon. Right? People are, what we're looking at doing next. We've got our hands around maybe some some typical, RAG chatbot use cases, and now we're looking to take it to the next step. And today's webinar is perfect for that because you can think of it as the first step beyond a RAG pipeline. How do you then get that thing to do something something meaningful after you've asked a question? So awesome. Thank you for sharing. I will go ahead and pull back up last couple slides before I step through here. So, before we kinda get to the demo, I wanted to go ahead and hand it off to Christian for a moment to talk about the thing that we're actually gonna build. And so I'll give it to you, Christian. Go ahead. Yeah. Thanks very much, and hello, everyone. So I'm really excited for today's session. I've given myself the challenge of build of live building a fully fledged agentic system, with a front end user interface and an entire rack pipeline in around thirty minutes, today. So, a fun challenge, but I think it will reflect the power of Dataiku, and really what our customers are leveraging, today. So before, actually going into the demo, I want to discuss the use case. So this is a system, that was actually brought to me by a customer that I work with very closely. And they actually had a very similar kind of stall pattern to the polls that we went through today. Right? So they've got a couple of different rag chatbots, implemented, but they wanted to go that next step. Right? Okay. How can we move into the world of agents, in a in a way which is relatively low risk, but potentially a huge amount of upside. Right? So in in the spectrum of, let's say, agent autonomy, this would be still very much human in the loop, but still the ability to perform certain actions based on reasoning. The use case that I have, I think, is very relevant for most industries. Basically, they had lots of documents in different databases, whether that's SharePoint, S3, Blob Storage. They actually had a combination of all of them that detailed very specific internal processes. From manufacturing processes or HR policies, IT policies. They estimated that they had around a hundred of these types of use cases that they wanted to implement. And specifically, they wanted to do this because they had a lot of specialists, let's say, subject matter experts that would spend between 5070% of their time answering these ad hoc questions. A huge amount of time invested in pointing people to the right documents, answering certain tickets, answering certain questions. And so what they wanted to do is, hey. Could we build a a Ragged chatbot on top of this, so we'd be able to to, offload some of these, these questions, and therefore save some time? But they didn't just wanna have a Ragged chatbot. Right? They they were thinking, okay. We wanna go that one step further. Could we have a a a a genetic system that could evaluate whether they the the right pipeline had enough, let's say, information to answer the question, correctly to the user. Right? So it wouldn't just answer it if it if it didn't know the answer. So essentially evaluating the answer. And if it can answer the question, if it does have the right documents to be able to provide the user with that question, it of course answers. If it doesn't, it creates a support ticket. In that case, they wanted to use ServiceNow, but that could be pretty much anything. For the purpose of today, we're actually going to use Airtable, but it could be pretty much any, support ticketing system that had an API connection in the back end. I think that is the background and I'm very excited to dive into it. Great. I will hand it over to you to share your screen. Perfect. As we mentioned at the start, if there's any questions, feel free to put them in the chat and we'll try and get to them. You should be able to see my screen now. Again, we're going to try and build this out live in about half an hour. Live building is always fun. We might run into some issues, but you'll have to watch me fix them on the fly here. I've already created a project. This, of course, is Dataiku for those of you who are not familiar with it. This is the front end user interface. We've got loads of different projects that we can access. You can see I've been working on some of our business solutions actually. Market basket analysis, demand forecasting, RFM rich customer segmentation. There's tons of use cases you can tackle with Dataiku. For today, I've got the document and ticket agent that I'm going to dive into. This is the project homepage. We'll skip straight into the actual flow where we're currently connected to a folder. Again, this folder could be anywhere. This could be a SharePoint. This could be an S3 bucket. This could be Blob Storage. It could be pretty much anywhere. If I dive into what we've actually got in this folder is we've got three PDFs. Very common PDFs. These are all synthetically generated PDFs, of course, for, in this case, Apex Industries. We've got our health and safety data. We've got our IT policy, and we have our travel policy. So very normal, kind of documents that most organizations, have. So what is the first step that we want to take here? Now typically, when when you're building a RAG pipeline, we do something called embedding. That's essentially taking these documents, and vectorizing them, essentially converting text to numbers so that when we ask a relevant question or when we ask a question, the system can pull the most relevant documents, based on that question. So this is a very common, approach. And I'm happy to say Dataiku makes it extremely easy for us to do, that. So what we can do is we can select document folder. And then here in the right top, we've got the embed documents, recipe. Now I love this recipe. It recently got released, and, essentially, it takes, some pretty advanced techniques and makes it very easy for us to use. By selecting this, we get a couple of different options. What's the name of the knowledge base that we want to create? We'll call this documents embedded. The embedding model we get to select, and in our case, we will just select four o here. And we'll go ahead and create this recipe, and we'll go ahead and apply this run here. Let me run you through, whilst this is running in the background, let me run you through this. Essentially, we have a couple of different techniques that we can use to extract the data from structured documents. In this case, we're actually extracting only from PDFs. But if we did have PowerPoints or docx format files, we're using Visual Language Model extraction, so in this case, four point zero. We're providing it with all of the documents and we're asking it to extract the most relevant data from those documents. Now what's cool about this is that we may have more than just text. We may have tables, we may have images. In the case of the customer, they actually had a lot of images in their documents such as symbols and signs, that they wanted to extract. So we're essentially, extracting all of the relevant, information from these documents, and, putting that into a into a vector space, essentially. You can see here we are actually done, running that. It it was as simple as that. And trust me, I built this without this recipe in the past. It can take a ton of time. Right? Like, trying to figure out how you're gonna process images, how you're gonna process text, different file formats, it can be a bit of a nightmare. The fact that we did that in less than a minute, I'm already calling a bonus here. We've got images folder. This is any images extracted, it will put this in the folder for us. And we've also got, importantly, a document or a vector a knowledge bank, I should say. And what we can do is we can go into this knowledge bank and we can start exploring it already. Right? So if we wanted to build a, chat interface on top of this knowledge bank, like a basic q and a chatbot, we can just hit that button there. It would be easy enough to to to spin that up. We can also test it in a Python notebook. So, anything in Dataiku that we're building, we can kind of interact with using Python. So if you are a programmer and you want to build a agent in a coding user interface, then of course we've got Jupyter Notebooks that we can do so. Let's actually test our knowledge bank here. First, we actually want to select which LLM we want to use. In this case, I want to use this one here. Let me just add that and we'll run through here and we will check to see that we're fetching the right information. Here, you can see it's similarity searching and it's gotten some documents for us, which is great to see. We've tested our knowledge bank, it works. We've embedded documents in the vector space, which is great to see. The next step from here is we, of course, want to start building out agents. Now, agents are great. They can have sequential steps that we've discussed already, but they can also have things called tools. Tools are a concept that we use a lot in the world of agents and they essentially allow our agents to interact or interface with the outside world or external systems or other agents. Essentially, they can perform certain actions very similar to how a human might use, might use a certain tool, whether that's Excel or whatever type of tool that they they may use in a day to day basis. And We essentially want to expose LLM to those tools as well. To do so, we can come over to the NavBar and we can go to Agent Tools and we will create our first tool. We're actually going to add two tools to this specific agent. And Dataiku comes with a set of tools which are very useful and very powerful. So we can see here the set of tools that Dataiku comes with, we can look up a record in a dataset. Right? So if you want to fetch a specific record in a dataset, we can do so. We can call another LLM or agent. We can append a record. We can send a mail or a message, and we can also use a traditional machine learning model and predict something using that, model, which is very powerful because it allows you to mix in agents with traditional machine learning to efficiently build some user interface for a user to interact with. In our case, the first tool that we're going to use is actually the knowledge bank retrieval. We just built a knowledge bank, so now we want to provide the agent with a way to interact with that knowledge base, to fetch documents if it feels that that's the right first step. We will go ahead and select this one and we will call this tool document retrieval. We'll go ahead and create this. Now, of course, it comes with an interface where we can select all types of different settings and how we want to interact with metadata and you can get very complex. You can improve diversity of documents here, of course, as well. In our case, we'll just select the knowledge bank, which is the only knowledge bank, that we have here. We'll go ahead and save this, and we can run a quick test. And we'll say, can you fetch remote working policy documents? We'll just test to see whether that's working here. Perfect. So we can see it fetched four documents because that's what I specified in the settings, that are most relevant to this specific tool. Now we will add the second tool here. We'll do a new agent tool and scrolling down at the bottom here, we've got the Create Airtable ticket. Now this is not a built in tool. This is actually a custom tool. Dataflow has, of course, always had the ability for you to extend the platform's capability if that's something that you wanted to do through plugins. This is actually a plugin that I developed myself and we've got a couple of open source plugins that you can see online how you can build these out. But very simply, it's essentially a function with an interface to the agent to interact with that Python function. In this case, it's just using the add table API to create tickets. Now again, this could be any support system. It could be ServiceNow. It could be pretty much anything. We'll go ahead and select create Add Table ticket here, and we will select, and we will call it the Add Table ticket tool. I'll go ahead and create this. It comes with a preset with API key, which is correct, the base ID. I'm happy with this. We'll go ahead and save. Then the next thing that we want to do is actually build the agent. We've just added a set of tools. Now we need to build an agent to leverage those set of tools. To do so, we can come over to the right bottom here. We can go to generative AI and we can either build a code agent. This agent was actually originally a code agent that we built out. Very simply put, a code agent just uses a line graph or a line chain or a framework under the hood that you can code out in Python, pull into Dataiku, and leverage very similarly how you would anyway. But in this case, it's a visual agent, which of course means that you can expose a lot more people to build these agents and of course document the processes and automate those processes using agents. In that case, we'll use a visual agent and we will call this documents and ticket agents. Christian, can I interject real quick? You can, certainly. Quick question. I've seen in the chat a couple of folks asking if we can zoom a little. Can you browser zoom maybe so some of that small text can be more That is a great one. We were at 125%, so we'll go to 150. Awesome. Thank you. Maybe even a bit more. Is this good? That looks better to me. Perfect. We can see here that we've actually created V1. It automatically creates V1 for us. Now we can select which LMM we want to use. Now again, Dataiku is very much agnostic when it comes to which LMS you want to leverage at which specific point in time to do certain tasks. So in this case, I've got my Azure OpenAI connection. I've also got an OpenAI enterprise connection. But if you had a locally hosted LLM like DeepSeek or Anthropic, you could, of course, connect to to those providers as well. We we allow you to remain very much agnostic, which means that, you know, if the new LLM comes out, you're not debating about, hey. Do we need to rearchitect our entire system? We enable you to very quickly switch that out, and leverage the latest and greatest. In our case, we will use four o, and we will just provide it with a prompt. Now I've already done some prompt engineering for us because you don't want to see me test out different prompts, but that, of course, is part of the process. That's really important to test out different prompts to see what works reliably. In this case, I said you're a question answering agent with specific restrictions. You cannot answer questions if you do not have the documents to back up your answer. When a user asks a question, you should always search a knowledge base for relevant documents. If If you cannot find relevant documents to back up your answer, you should not answer the user and instead create a ticket in our support portal for our support agent answer. Try to help the agent by drafting what you think could be a relevant answer and include this in the ticket. Always create a support ticket if you cannot answer the user's question. And once the ticket is created, use the URL and direct the user to this ticket. Right? So pretty simple prompt. You could of course get much more granular than this. But for the purposes of this use case, I think that will do well. Now what we can do is we can start adding tools. Right? So we've we we added the we we created those two tools and now we can add them to to our agent. The first one that we're going to add is, of course, document retrieval. We will just add a quick description so that our agent knows how to use this tool. This will say, hey, this is our document base. Use this tool to fetch documents. Then we will just come here and add our second tool, which is going to be our Airtable ticket tool. And we will go ahead and create this one here. And so now if we save this, we can again come over to quick test. For most of these interfaces, you've got the quick test option and here we will say, can you retrieve some of our remote working? Alright. So a a pretty straightforward ask. It's doing something very similar here, but it's going a little bit further than what we had with just the tool. Now it's actually making a decision on which action to take. Once this loads up here, it's going to, of course, our knowledge base and trying to see. I found several documents related to our remote working policy. Here's the key points. It details again, this is all synthetic data. This is not Dataiku's actual remote working policy. It fetches some relevant documents for us to leverage. Now, as you can see, this is not a visual user interface that you would probably want to give to your end users. It's a little bit cumbersome and not very user friendly. We want to take it, let's say, to the next step where we actually want to say, okay, we're happy with our agent, we're happy with how it performs, and I want to expose this to some end users to get feedback and see how it's working. We will actually come back over to the flow and now we can see we've got our little agent here connected to, of course, our documents, our knowledge base. Again, that data can be anywhere. You can connect to SharePoint wherever that data might be. We've created that embedding and now we have an agent with a set of tools on top of that knowledge base that can perform certain actions. Now we want to create that visual user interface. We can again go to the nav bar and we can go to Dataiku Answers and Agent Connect. Now, Dataiku Answers is a really nice visual web app, so you can essentially create it in a visual way and you can create a really advanced web app for users to start consuming LLMs and agents. We also have Agent Connect, which I'm a big fan of. We see a lot of organizations have multiple agents. Right? Maybe you have an HR agent, you have an IT agent, you have a Workday agent that can, maybe book you time off. Right? There's many different agents your organization might have, but you don't want users to have to go to different, user interfaces to leverage each agent. Right? Because that can become suddenly you've got 20 bookmarks of all of these different internal agents. And so Agent Connect, if you do have multiple agents, is a really nice routing portal that routes the user's questions to the relevant agent to, answer, the the question in the best possible way. So it's really cool. I'm a massive fan of it. But for today, we're again, it's a very basic or relatively basic agentic use case. So we'll just use the the built in Dataiku Answers, visual web app for this one. And we will call this document and ticket creation portal. Right? Call it what you what you may want. We are now in this visual user interface where we can essentially fine tune this front end user interface. We've got LLM and so now in addition to Azure OpenAI and OpenAI Enterprise, we've also got the agents. If you've got agents listed in your project, we can, of course, leverage them here. We'll go ahead and select this one. And now it's asking us to create a couple of different datasets for logging purposes and, of course, performance benchmarking purposes. So the first one is conversation history. This one is very important as it will allow us to, essentially check, all of the conversations that people have had with our agent. So I'll just call it ch conversation history here, and I will use our Snowflake connection here to create that dataset. So we've now got our conversation history, dataset and we will also create a user profile data. Oh, good. And so the user profile will store things like preferred languages or any type of user preferences that we may have. Now scrolling down a little bit further, we will also just give a nice system prompt here. Instead of saying, hey, you're a business analyst, which is the default one, we actually will say, you're a helpful support assistant, user internal knowledge base to answer questions. This is just the system prompt that is always included when we ask the LLM any questions. Then LLM for title generation, we will also just use four point zero here. We'll just use actually disconnection and that should be fine. Right? There's loads of other settings that we can change here, like display title, subheading. We can add example questions, so people often like to add the right example questions so people know, or get an example of what to ask. And Of course, you can also add your own custom branding. Most of the customers that I'm working with that have implemented this have their own logos and colors, so that it feels like a company, portal essentially. We'll go ahead and save this, and that will start spinning up the back end. And we'll go ahead and start running some tests to see if the agent is performing in the way we like. Great. The back end is up and running and now we can go over to the view section. We were in, let's say, the admin section. The view section, of course, would be exposed to the users. If you did have users, they would be able to come into this portal and interact with your agent or agents if you had multiple agents using Agent Connect. The first question we may ask is, can you give me some info on HR documents? The expected outcome here is that it's actually going to do the exact same thing that we ran in our test a little bit earlier. It will go to our knowledge base, it will find relevant documents, and detail that to the user. But here we can see, hey. These these are HR documents, policy review, data retention, health and safety. These are really good. I'm really happy with this. This is good to see. But now I wanna answer or ask a slightly less relevant question. Right? So let's see. So here we we're saying, can you find info documents about the secret Coca Cola recipe? Right? So this these documents don't have this. So we hope that the expected outcome here, the agent, would make a decision to say, hey. We actually don't have the the documents to be able to answer this question. So we were instructed to create a support ticket. So let's see if it does this. Nice. Here it says, hey, I couldn't find documents related to the secret Coca Cola recipe in our knowledge base. That's a shame. I'm very curious. But what it did do is create a URL for the user. We can go ahead and this is ad table. Right? But again, this could be ServiceNow, this could be anywhere, where it's created a ticket for me, ticket name, ticket content, the user is asking for information, documents about the secret Coca Cola recipe. This, of course, is a sensitive topic. But we did instruct the agent to actually draft the answer. Right? So I said, hey. The, the the AI will actually draft the answer for the support person that that's going to answer this question. And this might save some time. Right? Like, I don't have to tell tell the user that this is a super secret thing that we don't have the the documents for. So I'm actually quite happy with this. So maybe I set it to done, right, straight straight away. And, I've saved myself some time and the user has their answer straight away. We've gone over the process and again, I think we've done it pretty quickly. We've connected to our internal knowledge. We've embedded that. We've then created an agent with two sets of tools to be able to perform actions on top of the data. Now, of course, we could go much further than that. Right? We could connect to different agents. We can have more knowledge. We could, we could use traditional machine learning, in certain certain areas as well. But I wanted to show this because I think being able to do this quickly and exposing this to users so that they can start getting access in a low risk use case is very powerful. Certainly, the customers that I've implemented this with already are very happy because, of course, a lot of people are asking, especially in leadership positions, we need agents, we need to start cost savings, efficiency gains, and so I think this is a low risk, high value use case that can be implemented straight away. Now, of course, what's really important is to start monitoring the quality. The best way to do that is exposing it to users and getting feedback. It's also a little bit of, let's say, education to the users to say, hey, if you see a really good answer that you like, it would be super helpful to leave us feedback. And so if we really like this answer, for for example, this is very good. So what we can do is we can select that this was a good response, and we can, of course, provide more information and we will submit that. So we we said, hey. This is a really good, example, and that will then, feed into our conversation history dataset. So if you recall the conversation history dataset that I was speaking to, a little bit earlier, We will dive into this one. And so it will pull it, in this case, from a Snowflake database, but this could be any database. Right? Dataiku, again, very much technostic even when it comes to databases. So any database, whether that's a typical SQL database, Databricks, Snowflake, we can connect into that and leverage that. We can see here, we've got conversation ID, conversation name, we've got a couple of different things, filters, the answer, the sources, and the feedback. We can see, hey, this is a positive feedback. So maybe what we want to start doing is filtering the negative and the positive and start extracting learnings from that to help us prompt engineer, and even to provide examples to the element to say, hey, this is what good looks like. Right? So you you can add examples to to the agent, to the prompt. We've got that in one centralized place. Again, without having to think about it too much, we've got a really powerful dataset for us to use. Now, of course, this is an enterprise grade application and what's really important is to monitor performance over time. And so Datacom makes it very easy using the Evaluate LLM recipe. So if we select our conversation history and we scroll down here, you can see the LLM recipes that we've got access to. In our case, we use the Evaluate LLM, and I will create an evaluation store. I'll call it eval store. I'll go ahead and create this one. We'll create this recipe. And the input dataset format, we're gonna keep it as custom. But if you, wanted to use a specific one, you can do so. The task in our case is in fact question and answering. The input column would be question, the output column would be answer, and the context column would be sources. Now, Dataiku out of the box comes with a whole host of metrics that you can use to monitor the performance of your agents and your LLM applications. In our case, I will select let's see which one. I'll do faithfulness. Actually, I'll do answer relevancy. This is a very good one. So we can see here, focuses on assessing how pertinent the generated answer is to the given prompt. This metric is computing you computed using the question, the context, and the answer. It requires an embed LLM and a completion LLM. So we're gonna use what's called a LLM as a judge technique, where we're essentially saying, hey. This was the user's question. This was the answer, and these were the sources, that were provided as part of that answer. Tell us was this a was this a relevant answer? Yes or no? So here we define which, embedding LLM and which completion LLM. So we'll keep this one as I'll just set this one to this connection here. We can add custom metrics as well. If your team does have a specific set of custom metrics that you want to track over time, you can add custom metrics. I have a couple of clients that I've worked with that have added this. We also provide you that option here. I'll go ahead and run this. This will now look at our questions and answers and see, hey, how is this application performing? Now, in a real world scenario, of course, before we push this into production, you would want to automate this where you say, hey, I want to run this every day, every week to monitor the performance of your LLM application over time as users are giving feedback and interacting with it. We can see how many positive feedback responses, how many negatives, and we're also now using the LLM as a judge technique to say, hey, how relevant are the actual answers that, that we're providing? What's really cool off the back of this is you can perform all types of analysis on this text as well to maybe guide, subject matter experts on where the gaps are in, in the documentation. So the customer that I actually worked with, I asked them with the time saved, they estimated again, some SMEs were spending fifty-seventy percent of their week answering ad hoc questions. With that time saved, what are those SMEs going to be doing? Their plan was to take the questions that the agent cannot answer and document those processes better because now we know where the gaps are. If the agent can't answer it, that's a direct correlation with we don't have documentation in place to be able to provide users with those answers. It's showcasing where they need to work on, where they need to improve the documentation. I thought that was a a really interesting insight. And so coming over here, now we've got our conversation history into our eval store. And of course, the answer relevancy, in our case is zero, which is fine. And now we may wanna add additional, additional metrics. So if we say, hey. We also wanna measure maybe faithfulness, is a is a relevant one here as well. We can go ahead and run this one. And faithfulness is measures the factual consistency of the generated answer given the context. The metric is computed using the answer and the retrieved context. And again, it uses the embedding LLM and the completion LLM here. Let's see what we got. There we go. This one has run and it's also zero in this case, which is fine. So that's really what I wanted to go through today. I think showing how you can build an agentic system in, I think, what what are we at, twenty twenty seven minutes, so we're we're right on time, in a really efficient way, using a different set of tools, right, custom tools as well as the built in tools, I think is really powerful. And again, the customers that I'm I'm working with that have built this and implemented this are extremely excited with the possibilities. So maybe over to some questions. Awesome, Christian. Really appreciate you stepping through that. We will turn to some questions now. We've got quite a few coming in, so we may not be able to answer all of them. You can reach out to us afterwards if, if we don't get to your question. So first, we have quite a few questions around guardrails. So how are, how are the guardrails conducted? How can we make sure the right information is being passed? You did touch on this here at the end of your demo. I think a lot of those questions were asked before you got to, to the eval and the guardrails. But just to reiterate, those are kind of built into the augmented LLM. You can make sure that your answers are relevant, and and there's a lot of things built into the LLM mesh to ensure that we have the right types of information being parsed out to the end users and Christian I don't know if there was more you wanted to add to that Yeah Dataiku comes with with sets of guardrails so of course, the the, PII detection is a very, very common one. We also have toxicity detection that are built in. What's cool about these is they they're actually built into, built into the platform Dataiku itself. So it's essentially a guard before we actually even get to the LLM. Right? So PII, you may not want to send that directly to the LLM endpoint. So Dataiku can act as like a middleman to say, hey, that shouldn't be sent. Same with toxicity. Right? Maybe you don't want to send that at all. And we also have other guard services. So a very common one is CostGuard. And this is a very popular one where we have a lot of senior leaders that want to say, hey, I want to know down to the user, down to the use case what are the actual costs of using these LLMs because a lot of the LLM providers might make that, quite hard and cumbersome to find the answers to. Because Dataiku everything is order controlled, everything is logged, we can, to the user, in fact to the call, to the question, see the exact cost. So CostGuard, again, is a very popular service there as well. So, we'll make sure to share those. They are listed online, but very popular and very easy to implement. Awesome. Appreciate that. And we we also had a few questions around, the interface for responses. Right? Is it limited to the chat? Can we add other elements like graphs? I think, you know, we're not, generally, we're not limited to displaying text. So we can we can display images and other things. Yeah. I've got a couple of different customers that have image generation. Right? So, they want to do image generation. We can showcase that, and, you can showcase graphs as well. So it's it's, markdown compatible. So if you wanted to implement graphs, using URLs, that should certainly be possible as well. Awesome. There's one common one that I'll address as well for everybody, which we showed this live. So, obviously, it's real. Which version? Dataiku thirteen point four has the latest version on all of the, agentic items. That said, some of the the retrieval augmented generation steps that Christian showed are available in earlier versions as well. And there's a link in the chat to kind of the, the latest updates for Dataiku if you want to get more info on that. I'll step into one other question, Christian, that you might be able to help with is, we've got a few questions about people asking what type of underlying LLMs can we connect to? So some people may have, the desire to switch between providers like OpenAI or Anthropic, and some may have, like enterprise c four compliant LLMs that they wanna connect to. So maybe we can speak to how we can connect to those. Certainly. Yeah. We can connect into, of course, all of the common ones, but even if you have LLMs that are hosted locally, then of course we can connect into those as well. Or if you've got them maybe on AWS Bedrock, you can host custom Lambda models, we can connect into those. So we're very much agnostic when it comes to which LLMs we can, we can leverage and and connect into it. That that's one of the key value propositions. Right? This technology is moving so quickly. I think every two weeks, this is a new LLM that comes out and then everybody's like, oh my god. We need to use that. So we want to enable teams to be efficient at evaluating, of course, the new LLMs to make sure that, hey. This is in fact for our specific application performance. There's also the question around cost versus performance. If you've got an application like this, maybe using the largest model, the performance you see might not be worth it. Maybe four point zero Mini would probably be more than enough in this instance. Having the ability to say, hey, let's this week or maybe perform AB testing, let's have half of the people use four point zero, half of the people use locally hosted model is also a huge proposition as well that I've seen leverage. To answer the question, we can connect into pretty much any LN that you've got. Awesome. Definitely one of the strengths of building Dataiku is the ability to swap out those underlying technologies, and we will be doing some more, I would say, content and webinars on that topic specifically as that space is changing so quickly. Did get a couple questions about, who can use what. So if we think about creating an answers chatbot like you displayed, does somebody have to be a Dataiku user to interact with that versus if somebody wants to, actually, develop it? Like, what kind of licenses are we talking about? Yeah. We've got a couple of different licenses, of course, when it comes to the platform. To consume what we've built, you would want the AI consumer license. If you want people to come in and access the application, and then depending on your package, there's a couple of different license types that you may wanna leverage to actually build out the, the agent and the and the ad elimination. My my best advice there is, speak to your, representative, typically your your account manager, and they'll be able to guide you to the right license to use for you. Then related to that deployment question, what if they wanted to deploy this somewhere besides the EchoAnswers? What's our ability to extend into other types of applications? Yeah. Another great question, a very common one as well. The LLM mesh is really cool because at the back end, it's essentially an API. If you wanted to connect into the Visual Agent that we just built using an external web app, Maybe you have a really great front end, development team or you've already built a front end application that a lot of the people are using. Then you can just directly connect into the agent using the API. And then, of course, you've still got all of the monitoring, the oversight, the guardrails, in place in Dataiku, to be able to monitor that, but of course, connecting into the external system. The same goes for the knowledge base. If you wanted to just expose that knowledge base through the API, that's certainly possible as well. Yeah. Alright. Beautiful. Let me see what other ones we have not yet mentioned. Yeah. A lot. There's a lot of good ones trying to group them how it makes sense, to respond to most of these. Oh, here was a good one. Which vector databases can we work with on Dataiku agent recipes? That is a good one. It's constantly changing. So let me see. I had the documentation up, actually. We've got ChromaDB, Faeth, Pinecone. Those are some of the ones I can see at the moment, but they're constantly being added. So I would say the common ones are certainly there. If you've got a very niche one, then again, reach out to your solution engineer and we can see if we can connect into those. I think we'll probably be able to post the documentation for the vector source. Maybe Sylvain we'll be able to do that, in the back end. Yeah. We can post it into the chat. There's, you mentioned most of them in documentation page themselves if you go to look. There's Azure AI search, Elasticsearch, OpenSearch, Pinecone, Vertex, VectorSearch. So we'll share that link for others to be able to explore. Okay. Let's see. What other questions have we not answered yet? How about, you you touched on this a little bit in the demo, Chris, but there's some questions around, agent observability. So, like, what are our options to observe the agent as far as the metrics for accuracy, but maybe also seeing what the agents do as we give them more steps? Yeah. It's a really, really good question. We've just recently released a Traces visual web app, which I'm really excited for. I don't have it right now, but I'll link to the documentation. But essentially, it's a visual web app that allows you to look at the traces of the LLM and see a step by step approach of how that LLM is interacting with the different tools, with different LLM calls that it's, that it's using. Everything is logged. So that's that's a really key thing as well. Everything is logged. The traces, plugin just uses those logs to visualize that for the user. So, I think we've got we've got a we definitely have documentation for it, so we'll link that. I think probably in an upcoming webinar, we'll be able to showcase that. Awesome. I'll answer a few more questions before we wrap up and talk about the next upcoming webinar. Did get a couple questions about LLM costs. So the way to think about this, this this question is, do the LLM costs, are they in addition to the platform cost? Is this understanding correct? And I wanna I wanna answer that question directly, Christian, but I also want to reiterate to people how we help them make better cost decisions, with regards to their augmented LLMs. So the first the first part of that question is, are the LM costs in addition to the platform cost? Yes. As you're using different underlying LLMs, you'll have cost for the calls to those LLMs. But, Christian, maybe you can mention how, folks can make better cost wise decisions. For instance, prompt studios where we will surface the different costs, the different queries, anything like that. Yeah, certainly. Again, Dataiku, the agnostic to which LLMs you can use, we make it very transparent in terms of what costs you're incurring. Though there's a lot of the time the question of, okay, should we use LMs? Is it the right LLM? Which LLM should we use? Within prompt studios, we actually display the cost per thousand queries based on the input tokens that we can actually estimate. You can test out different prompts, different inputs. Maybe you've got a couple of different columns or lots of text that you're inputting, and so we'll be able to provide you an insight to say, hey, If you're gonna run this on a million records, this is likely the incurred cost, for for your specific project. But, again, going back to cost guard, it gives you a really granular per user per project, approach to say, hey. These are the costs that you're currently incurring. And, of course, as a solution engineer, I wouldn't be be doing my job without, mentioning value. Right? That's the, let's say the antithesis due, to cost. So making sure that you've got, the right value tracking in place, which is, again, something that, your representatives, your solution engineers, your account manager will be able to help you with. Identifying, hey. This use case that we're implementing, so this specific one that we went through today is efficiency gains. Right? So can we estimate we did this, of course. How many SMEs are how much time of those SMEs have we saved? What are they now doing with that time? And then attributing cost to that to say, hey. Or attributing value to that to say, hey. The the cost of this specific application is x per year per month, but the value achieved is y. It would make it very easy for you to perform that RI calculation and also to be able to make adjustments to say, Hey, if we do need to reduce costs or optimize costs, this is how we can do so using using the platform. Perfect. Thank you. Maybe the last question before we wrap up, that I'll highlight is, there's a question around how does the oh, I'm not sure the agents we build on top of it. How do they handle, updates to the underlying knowledge base? How should we think about that as that underlying knowledge base changes? Yeah. Again, a very good very good one. In a recent release, this was actually optimized. So, Dataiku actually tracks the metadata of the documents. If a document is updated, then it will just update that specific file in the knowledge base as well. It means that every time you run it, you don't have to rebuild the entire vector base on all of the documents. It will just select the documents that have been updated and, and of course, update that in your knowledge base and you can automate this. That's the beauty of Dataiku. You may say, Hey, we want to run this every week. Every month we want to automate the updating of our knowledge base so that users always have access to the most fresh information. Awesome. Beautiful. I know there's a few other questions we didn't get answered. If you go ahead and reach out to us, we'll send out the recording, all this good stuff. But if you want to reply and reach out to us on other questions that we didn't get answered, we'd love to to help answer those. Specifically, today, we're really looking at building a relatively simple agent. And, Christian, really appreciate you walking us through that. And this is a good place to start, especially we as we noticed in the polls, in the beginning of this session, a lot of people have their arms around kind of like this rag chatbot use case, but maybe are looking to step into agents next. So this type of use case is a really good way to go from step one to step two. But if you want to extend your agents to more capabilities, more use cases, you want to do that with tools typically, right? Give it more firepower to be able to do things like search the web and send emails and all this kind of stuff. So one of our upcoming webinars in April we will we will talk about tools that'll be like the next step in our hands on series, and we'll we'll build out a little bit more capabilities to make the agents even more powerful. But the the next upcoming webinar in the series that I'm gonna draw your attention to, I'm excited about because we will, have a webinar with our CEO, Florian Duarteau, where he'll will, he'll debunk the AI agent hype. So we'll talk about separating this from reality so we can focus on driving real value with agents. I think this is gonna be important so we don't waste our time building in the wrong direction or building something that's not gonna be extensible to the rest of our ecosystem. We're very focused on helping customers make sure they can build things that will really drive value, get out of the lab and into the real world. So excited to have Florian talk to us about that on March. The QR code is up here if you want to register for that now, and, keep your eyes for the following webinar to that one, which will be around tools for your AI agents. So with that, I wanna say thanks, Christiaan, for, walking us through that awesome demo, and thank you everybody for joining us today. This was a really great conversation. I'm looking forward to the next one, and we'll see you next time. Thank you.