Name: What’s New in Dataiku 14.5: Bring AI Into Everyday Workflows
Uploaded: 2026-05-07T10:16:19.222Z
Duration: 48 min 28 s
Description: What’s New in Dataiku 14.5: Bring AI Into Everyday Workflows

Transcript for "What’s New in Dataiku 14.5: Bring AI Into Everyday Workflows": Self serve. Okay. What happens if I So now we should be live. So welcome to our second webinar session, this time about the 14.5 update. We will wait a few more minutes so that everyone can join and then we will start for a short introduction and go through all the cool releases that we had with 14.4 five. So let me put that information also in the chat. And if you're looking for the chat, it is on the right hand side of your screen. I know you're looking at me, and it looks like I'm going left. So we could do it right hand side when you're looking at your screen. Noise. Maybe a first shot check. Can you can you see my screen? Short thumbs up in the chat would help us a lot to make sure that everything is working fine from our side. Perfect. Thank you very much. Perfect. Nice. K. The number of attempts is growing and growing. That's good to see. As I mentioned in the chat, two more minutes, then we are good to go. Okay. In the meantime, for everyone who has already joined, we will start the session with a short poll. So when you go to the to the chat window, there is a window called poll. And please enter the topic that you are most interested in today. So all of the topics mentioned in there are things that are released with the fourteen point five update. And depending on your responses, I can spend more time on, the parts that you are most interested in. I will repeat this in one more minute, but for everyone who have already joined, they can already answer the poll, please. K. Few more seconds, then we will start the session. Okay. So welcome again to our second feature update session. This time, it is about our 14.5 update that happened in April. And my let's start with a short introduction. My name is Tommbley Koch. I'm a customer success engineer based in Munich, and my task is to show you the latest feature updates and support you that you know what cool things were introduced in our latest update and together with me is Harry. Harry can you introduce yourself? Yeah, sure. I too am a customer success engineer. I'm based in Berlin, and my role today is to support any questions in the background. Any questions throughout the webinar, please post them in the Q and A. That is much easier to follow along. It doesn't matter. You can also post them under messages, and I've just written that within the chat. If you cannot see the chat, it's on the right hand side of your screen. Ah, you can't hear me? Dominic, can you hear me? Yes. I can hear you, but you have to unmute yourself. There is, I think, a mute button for can of you is you try testing. Test. Oh, am I again? personally muted? Okay. So. we can hear you well. Okay. Now it works. So Harry was I think he was still muted because I was the main, That's. it. Okay. Harry, can you redo the the intro for your part? Yes. So, Annika, just very quickly, I I'm also a customer success engineer based in Berlin, and today, I'll support with any additional questions you may have. So any questions that are posted within the chat, I will be there to support those questions and answer them as best as possible. I highly recommend using the q and a. This is a tab on the right hand side of the chat interface. It says q and a. It's next to the poll, And please, fill out the poll if you can. This gives Dominic a bit of focus on what we can all show you best. Okay. Perfect. Then I will take over. So today, I'm going through our Dataiku release in a box document. And before I start with this one, it's always, I think, a good way to show you where can you find these documents. So when you go to our what's new in Dataiku page and you scroll down to the bottom of the page, you will see there is one of these documents for all the releases since 14.o. So if you are interested in all the things that I'm showing you today or if you're interested in the things that were added in the latest, in the last updates, go to the site. You can download the PDF file that I'm showing today. And yeah. So it's a very good summary to to see what was added as a latest feature to our platform. Okay. Then let's jump into this document. So today, we have multiple cool things to show. One is about our Gen AI and agent capabilities. There are several, additional features about analytics and insights and many, many things for your coding experience. So if you are a hands on coder, bear with me. We have very cool things, coming up in a few more minutes. Then let me show let me look at the poll. Interesting. I can't see the result of the poll. Can you please send me okay. Here's the poll results. Go to the poll. Stop. One moment. Okay. I still can't so the poll, I think, is still up and running. So please, send me a short screenshot. Okay. Now I can see it. So most people are interested in our service AI assistant deployment, agent logging, and also code studios. Interesting to see that they're only yet the lowest vote for the extract fields recipe because from my experience, that is one of the features that were was mostly requested by many of my customers that I'm actively working with. And to be honest, I think that's also one of the core strengths of our 14 dot 5 update. So what is the extract fields from document that I'm talking about? It is the opportunity to extract information out of unstructured text. For example, PDFs, Word documents, also images like PNGs, and it can also work on Excel files. Excel files officially are not supported because sometimes the structure of Excel files can be a little bit messy and difficult to interpret from, by LMS. So what can you do with this extract field from documents recipe that was recently added? You can upload documents, and you can give a description what part of the document do you want to extract. And then you get a clear table with all the information that you need, and you also get information if something didn't go well. And that's a huge game changer for everyone who did this before because in the past, you needed four recipes to do the same. So you have to extract the, text first, then you have to up cert the data in order to, prepare the data in a in a specific way, do a prompt recipe in order to extract information, and then do a prepare statement on top of it. So this this recipe makes your life way more easier, and that's also where I want to start the first demo to show you what how does it work and how cool this feature is. So now you should be able to see my flow. In this flow, I've added four test documents, for example, a PDF file, but also a Word file. Okay. I can't see the the preview in here, but it will show up in the in the recipe and also m p g files. So we have different kinds of documents, and you can already see here by comparing this invoice with the AWS invoice, they have a totally different structure. And that was the main challenge in the past to be able to extract information when the structure totally changes. Let me show you how this works. So I click on the folder which contains my files, click on the extract content, and now we have two options. Either I can extract the full content and get the text that is has been part since several releases, or this one is a new one, the extract fields, where I can extract specific fields out of the document, give a name to the output dataset, and here we are. So the first thing that I need to do, I have to choose which LLM do I want to use to extract this information. So we are using a visual language model, so the file is sent to the LLM. And given with additional instructions, it will give you back the, specific fields that you requested for. For example, I can choose GPT five minutei. So that's the first important decision. Depending on the model that you choose, this will have a huge impact on the result that you will get. The second thing that you can change is you can customize the extraction prompt. Most of the time, you can be fine with the default one where it says, okay, extract the fields, respect the given schema, so we will enter the schema next. And also very important, it should not make up anything and give us a null value if it doesn't find the information. That's the default. If I want to improve my, my results, I can always add additional information in here. For example, say to him, okay. That's invoices with a specific structure and, maybe also tell them if it's another language than English, that's also helps. So usually, the best way to do it is start with the default, figure out, is it working okay? And if it's not working, then, you can still deep dive and add more information in here. So let's have a look at, for example, the PNG file. How does it work? So now I have to give information. What do I want to extract? For example, I want to have the invoice number. This is a string. I can also say, okay, if it's a number, integer date, a boolean, or an array. I will show these features in a few more seconds. Let's also extract the supplier name, and I'm interested in the invoice date. This time, I have to select the date column, which is important that gives me the information in the right format. And now the difficult part. How can I extract that information where we have multiple entries? And that's where the array comes into play. So I say, let's extract the invoice items. This is extracted as an array. And here I need the item name, quantity, and the unit price. I'm also interested in the total price. And now let's just test it out. I will change the unit price to a number and let's see how it is going to work. Let's wait wait a few seconds, and then we should have a first feedback how well my instructions work. And here we can see, okay, it all of it is green, so it found something for everything. But I might change something. For example, for the total price, I got the information, a combination of the price and also the currency, which might be difficult later on to work with. So maybe it makes sense to split it into a currency field and the total price field. But overall, this one went pretty well. At this moment, I was only running it against this one example, so I didn't waste any additional tokens by sending everything to the, to the LLM and then maybe figuring out, hey. It's not working as intended. Let's try it on this one. So does it also work on the AWS file? And here, surprise, it should not find everything. Let's see what happens in a few seconds. Okay. We have an issue. It is orange. It says it only found partial information, and the issue is it didn't find the quantity, which is no surprise because there is no quantity in here. So we are only listing different services and also the price. So it's totally fine that it didn't gave me that information, but it's good to know because later on, when I when I run this whole recipe, extracted as intended by the schema, which is important because then in the end, I can only focus on these, and these entries where something is not going well. So everything run through. Let's have a look at the results. And here we can see here, for example, are the entries where it says, okay, something didn't go well. I can see, okay, for the invoice items, there was an error. The quantity was not found, and you can see the quantity field is, in fact, empty. And everything else so for all the other columns, it worked fine. Here, it didn't find the invoice date, but I know it there is no invoice date in this, in this mock up example in here. And also the cool things, I also tested it out with with non readable, PDFs. So when you go back to the PDF, here you can see I can select everything in here. So it's easier for the, for the VLM to extract what is in there. But here, I can't do this. So it's it's an image. I can't select the text. So here, the LLM is doing most of the heavy lifting in order to extract the information. And, yeah, that's our first, first new recipe. And, as I mentioned before, many of my customers have asked about such a feature, and it should make the life way easier to work with unstructured data because extracting information can be really tricky if the or it was really tricky before because the the format of these different invoices can look pretty different. Good. So far any questions on on that field extraction recipe? There is a question from Kelly, which I'm getting to answering on, which is if we have several 100 documents and disparate data that, you need to extract from. So you. have no no idea, have no way of knowing how each document presents its Perfect. Totally works the same. So you usually go through you you create the extraction c schema based on all the information that you want to extract out of it. So they have at least to have a similar, content. So I shouldn't, put invoice and also policies where I want to extract something, in in the same document extraction. But so it works totally fine if there are thousands or hundreds of thousands documents in here. You can test it with a handful, run it through, and then figure out, okay, was it fine? Are there any partial information missing? And this can save you lots of time, lots of manual labor that happened in the past, and you only have to focus on things that were not able to be extracted automatically. So from my point of view, a real game changer when working with unstructured data. Okay. So if there is no additional question on that one, let's jump back to the slides. And here we are with agent chat. So what is agent chat? Agent chat is one of our two solutions that we that we now offer. So usually, when working with agents or with chatbots, there are two distinct needs that you have. So either you want to quickly qualify if your agent is working and you want only to roll out this one specific agent. And there is the the second need that we see many times is that managed by the IT team. Someone wants to have a centralized place where you can engage with all your agents that are rolled out to the whole company. And for this for this second solution, we have the agent hub. And agent chat is built on the same code base and, is only here for a single agent focus. So some of you might think, okay. Hey. That's what Dataiku answers is for. And, yes, you are correct. Agent chat will replace the Dataiku answers in the near future. So similar to to Agent Hub and, Agent Connect in the past, we we had running both of them in parallel, but this agent chat has now the same look and feel, also the same database behind it as agent hub. So any improvements coming to agent hub will also come to agent chat. And yeah. So it will also make your life easier to to test, for example, one specific agent. So that's, in my opinion, also the biggest strength of agent chat. You can use agent chat in order to spin up a chatbot without any IT needs, and that's also why we implemented it because many of our customers who work with agent hub, it's difficult to use agent hub to test out a new agent. So that's what agent chat is mainly for. Any questions on the agent chat capabilities? K. I don't see any questions. And then let's jump. to the sec yeah. sorry, There. is one. There is one question. What is the difference between this agent chat and some traditional chatbots such as Copilot? I will also be writing my answer in the messages too. Okay. So with Copilot, you have your yeah, you you have built an agent outside of Dataiku, which you can also include in agent hub. Agent chat itself is built for using agents that you have built inside of Dataiku, where you give the agent access to your data, access to your machine learning. And that's also, from my point of perspective, the thing that makes agents more powerful most powerful. So agents in the end are only LLMs that get additional, additional skills by giving it tools. And so far, one of the most besides giving access to your data that it can have additional information that it otherwise doesn't have, giving it access to machine learning tools, that's that's a big strength of these agents. And agent chat is just the interface to to chat with it. Another way to consume the agent is build it inside the workflow and give it a task. But usually agent chat is the way how a human can interact with the agent. I hope that answers the question. Okay. Then let's jump to the next topic, and this one is about analytics and insights. And this is now for our enterprise only. What is process mining about? So maybe some of you have already heard about our process mining, business solution, which is still up and running. But setting up this business solution can take some time, and therefore, we have this new release, which is an application. And it can or it gives you the capability to look into process logs. You can visually go into them, figure out what processes are you running through. Is there any, any blocker? Are there any redundancies happening? Have we any cycles? So it's way easier now to understand if there is any bottlenecks or process performance gaps. Why is it important to look at these process logs? Because in the past, many times, these process mining things happened by doing interviews with the team. But then, usually, you only get feedback how the process should look like. In the end, by looking at the process logs, you will see the hidden things. Bottlenecks they do were not aware of or parts that workarounds that occurred because some part of the process is not working as intended. Herefore, I also have another demo to show you live what the process mining is all about. So for the process mining, as I mentioned, it's a separate app compared to our business solution, which maybe some of you already have up and running. Here we benefit from from a nice UI and also the the the option to to really drill down, into into the flow. So first, one thing to start with is the datasets. So I uploaded already two datasets to to have a look into it. Let's give a short preview. So here, important things that these datasets have to include is we need an order ID. So we have to group, for example, and say, okay, that is one process happening with the same order ID. We need either activity names or, for example, you can also, verify who is responsible for one specific task. Another thing is also which is mandatory is the event time stamp. And then in addition to the event time stamp, optionally, it helps to include the end time stamp and also an event sequence. Why is this information important? The event time stamp helps us to calculate the duration to figure out how long this one specific activity take, and the event sequence is here to break ties. If it happens that two events are happening at the same place, we need to have an order in order to create our flow. Everything else which is added in here can also be included in our flow and then used as a filter option. And that's what I want to show you live. So I already set up a study, which I'm going to load. And here, that's how my initial view looks like. So I have 9,600, ecommerce flows. So that's all the different flows that you can see. So we have a few deviations or different paths at the beginning, then everything stays the same. And in the end, we have several results. For example, the delivered was an exchange order created or a refund that was issued. How can I deep dive or how can I read information out of this flow? So usually, the darker something is, the more observations are in there. So you can see the left pass path is way darker than the right path. So here we have most of our observations. And when I zoom in, you can also see how many observations are in here. So we see it's 402 compared to 9,200. What else is important? We see, for example, here is a third pass opening up with a fraud check, but this is only a light blue. So we see 154 cases. And something else, by looking at the time, reveals what are the time consuming paths. So we see this is a path that not many users take, but is way more time consuming than other steps in our flow. And here later on, you will see there are some very dark brown colors, which usually happens, for example, when you return something because then you have to wait until the item, came back to the warehouse. It makes sense that it takes several days because it has to be, yeah, physically delivered. But if something like this happens in a workflow where it shouldn't occur, that information is really valuable. Also, now let's do a drill down. So what can I do? How can I interactively, use this, flow in order to get additional information? So just that let me click on the fraud check and say, let's filter it. And now I have a new flow, which is only for the 154, flows that went through this, this fraud check activity. And now we can analyze them, figure out, okay, the average for them is six point three seven days. We can also see by week how did it change. So we can see some ups and downs, but no huge changes. This visualization helps you to identify seasonal patterns patterns. We get an overview what is the outcome. So even with the fraud check, most of the things are delivered in the end. Some orders are canceled. Sometimes we have a refund. And in addition to setting a filter by clicking into it, we also have the option here to, to add, for example, custom filters. For example, when I want to look at the, the custom region, now it allows me to compare Europe, North America, or APEC against each other. So I can really use this to to, to get a good overview of my of my process path really fast. Why is this important that I can analyze now these log files? There are other tools out there which are perfect for the default processes. But here with the process mining and analyzing the log files, you can analyze any process where you gather log files. And you can also use this as a starting point for further analysis. So I can also go on the one level lower to every case. So what happens here? I have one entry for every case. I have them also grouped, by activities. So I have, for example, 18 cases that only have four activities. They it's a very short one. Why is it short? Because it ends with an order canceled. So the fraud check figured out, hey. Something is not right. So I immediately cancel it. And here I can deep dive into every of these, every of these, separate observations can figure out how did it go. And if I'm interested on further analysis, I can just click export cases. It will create a new dataset in my Dataiku workflow, and that's where I can do then my analysis on top. And that's also where the strength of the system comes into play. I have all the power of Dataiku in the background here to analyze also my process mining. That's it about the process mining part. Any questions on this topic? Okay. So far, I don't see a question then let me jump back into the slides. And then our next topic is the coding experience. So what can I tell you about coding experience? We had lots of updates in in the last, in already in fourteen point four, in fourteen point three. And this is, yeah, the the next, evolution for the the code studios. In the past, we added I think it was '14 dot three that we added codex and, the opportunity to use OpenAI, open code inside of Code Studios. And now we also have cloud code. So nowadays, everyone is talking about cloud code. You can also use cloud code directly in Code Studios now. You can run these things through LML mesh for most of the, LLMs. I think Cloud Code at the moment is not available for this LML mesh option. Same for Gemini. So Gemini will also be added soon, and it makes your life way easier when you are programming everything inside of Code Studios. Why is that the case? Because you don't have to to have your second window open where you are running your where you are chatting with your agents. You have everything in one interface. The agent has access to all your code and not the things that you just copy and paste in there. You have the security. Everything stays in the system. You are not copying anything, outside of the system. So for everyone who is hands on coding, it makes the life way easier. And if you are using the LLM mesh, that also gives you the opportunity to have cost control cost control on top of it to make sure that you are not, spending huge amount of tokens, and also, the audit at guard rate capabilities. But that's only one of the updates. Another one is especially interesting for everyone in here who is actively developing web apps with the help of our code studio or in Versus Code. In the past, you always had to run your code, open a second window, figure out what did it change. Now you have a preview window right in in your code studio, which may will make your life also way easier to figure out. Did it have the right the intended effect? So I don't have to switch windows anymore. And it also so as mentioned here here, you can use, codecs, cloud code, open code in order to, make your life easier also with the web app developing. And in addition to interactive debugging, you also now have the option to set breakpoints. Doesn't sound that important, but for everyone in here who already has to debug some code, it can be a lifesaver because it you can then jump line by line, figure out what didn't go well. Otherwise, you have to print out, the the status of some variables in between. Does also do the trick, but it's way more cumbersome to do so. So the breakpoint options, make also your life as a coder easier. And, again, so the third update on the code, on for for the coders in here. So we now have code agents testing and debugging included into Code Studios. So that was also included, I think, in 14 dot three that you are now able to, to code your age or to to write your agents in Code Studio directly. And here again, we have the the option to interactively test the changes, also add breakpoints in here. You can write unit tests based on auto generated Python files. So the whole process in order to create an agent with the help of code is now way more convenient and, especially the testing part, is way easier to do now. And I think that's our free updates on on the code front. We have a question on cloud code if it only works with the API keys. So in that case, I think you have to you you have to use your as I mentioned before, cloud code at the moment is not included in the LLM mesh. You so you have to use your own API keys in order to run it. Exactly. I also wrote here that it supports with the external OAuth direct API key or fetching an API key from an anthropic connection in DSS. Okay. Thank you, Harry. Good. Any any more questions on these code specific things? I think there are some some other code things mentioned later on in in the miscellaneous part, but these are the the main options. Okay. There is a question another question from Sala. I have built a custom agent in Versus Code. Is it possible to build to share this with our users so they don't have to create the custom agent from scratch for every new Versus Code instant? What do you mean Yeah. Go ahead. okay. The I I'm currently thinking what what do you mean by custom agents? So if you if you have if you are writing a code agent, everyone else should be able to to also work on it. Maybe the important thing is what what are you meaning by custom agents in here. I think for sake of time, I can go ahead and answer this one. Okay. Perfect. Continue on. Also, I'll yep. also very important if you if you are using, for example, skills for for agents, Now they also when when you use it directly in open studio, you don't have to reenter them again and again. Now they are also, you can see all the skills that you have used with your instance before. Okay, perfect. We have another one, totally forgot about the auto generated Python code documentation. So usually, especially with larger projects, the global libraries can can grow bigger and bigger. And now you have an searchable auto generated API documentation. So you have one point of, one, single point of truth where you can go and figure out, okay, what do I have in my global libraries, which also makes the life of a coder way easier, because he knows where to search for this information. And, otherwise, upfront, you had to to really know, okay, where where to deep dive. And another one on the SSH keys. Now you are able to create and enter SSH keys as a user directly in the system. So no need to to create them outside. It can be automatically create created by generated by DataIQ, filled in by the user, and then you can, for example, connect to your, for to your git repository on the fly. And, that's this can be done, at your personal credentials. So just go to your profile, go to personal credentials, and there is now a new field to enter existing keys or to generate new keys on your on the fly. Good. And that was already the main update. So now now we have a list of additional things that, didn't get a specific slide, but still can be very important. For example, we have now an agent tool, which is, able to create artifacts. So you can you don't only have to create text. You can also create images and work with them. So there is a new tool added to the list of agents. The second one was also requested really often that you can already leverage knowledge banks that were externally built. So far, you always have to create your own knowledge bank again. But if your company is already working with a knowledge bank, for example, Pinecone, Azure, Milvus, or Elasticsearch, you can use this and build on top of this existing knowledge bank, especially with larger ones. This can save you lots of time. Then agent logging, that was one of the things that were was requested or according to the poll mentioned, it's really important. And, yes, in the past, we have or we have several areas where an agent can be triggered. An agent can be either triggered, for example, in the prompt studio. It can be triggered in agent chat, in agent hub. You can trigger it directly in the flow. So we have different, areas where these agents can be triggered. And now you can set one default place where all these loggings are where all the logs are stored. So you set this up once at the beginning, and then every interaction with this agent is is stored. You can also say, okay. If there is one, for example, for prompt studio, you don't want to store this information, you can also do so. But we now have one one place to go to in order to see any interaction that happened with the with with the agent. And this one can also be reused with the agent eval. So this these two points are connected to each other, and we also can work out with multi turn. So what do we mean by multi turn? Most of the time or in the past, agent eval was only working with one interaction. So I am asking one question, nothing more. And now there's also the option to, to do an evaluation on multi questions. So if you if you have asked one question and then right after another, you have another question on top of this answer that the agent gave you. And therefore, this also means we have additional, metrics that can be used in order to figure out, okay, is the agent doing its job? And from my point of view, very important to set up this agent eval to make sure is the agent really doing, what it's intended to. So that's something that will come up in the near future with agent management, as an outlook because many at the moment, many or many agents are only monitored if they are up and running. Now with agent logging, agent eval, you are also able to to verify is the agent doing its job. And then with agent management in the near future, we we will be able to do everything with monitoring, figuring out is the agent doing its job, and that not only for Dataiku agents, but also for other agents in the field. Talking about agents, there are some additional things for agent hub. I think French and Japanese is not that important for the audience or at least that's my assumption, but we also have support now multimodal outputs for agents. And one more thing on the AI ecosystem, that was released beginning of the year is the Databricks Lakebase. And now you can natively connect to these, you can create a new connection which directly connects to your Databricks Lightbase system. And that's it from my side. So let me look at the chat again. Are there any more questions on all these smaller things that I just presented in here? Okay. I think Harry okay. Harry already answered the question from Kelly. Okay. So now we have one more poll. So this is the second time that we've done this webinar. So please give us feedback, if you are likely to recommend such a webinar to your peer. It helps us a lot to figure out, okay, shall we continue with the series? Also, one important information. This, webinar is recorded. So if you want to rewatch the session, you can go to the link, and watch the video on demand. You can share this video or you can share the, the link to the session with your colleagues. They can also just enter their email address and get access to this on demand, video. And from my point of view, that's the best way to showcase these new features to all of you. If you are interested in a specific deep dive on one feature where you say, hey. I really want to do an hands on on this, or I have additional question. Reach out to AEs. They will reach out to Harry and myself, and then we are happy to have a deeper deep dive on one specific, feature that we just introduced. And that's not only about 14 dot five. If you are interested in any older feature, ask your AE, and they will and then we will have a session on also one of these older features. Okay. Then please take take one minute to give us feedback on the session. As I mentioned, that helps us a lot. And then thank you very much for attending the session. Thank you. There is also a request here. Can you please repost the link to the slides? I believe is that already under the messages? I just got the email, the the the information that we will send the slides. in, via email, and I will also, add this information upfront so that you know where you can find the, all these release in a box files, because from my point of view, they are, at the moment, well hidden, but we are working on on on an easier positioning on the website. Okay then. Thank you very much for attending, and hopefully, we will see each other at the fourteen dot six session. Thanks, everyone. Thank you. Bye. bye.