Video: End-to-End XOps With Dataiku | Duration: 3632s | Summary: End-to-End XOps With Dataiku | Chapters: Welcome and Introduction (5.3599997s), Getting Started (96.635s), Agenda and Introductions (194.9s), AI Operations Challenges (287.25s), Operational AI Challenges (380.185s), Efficient AI Scaling (514.715s), Unifying AI Operations (650.19995s), AIOps Evolution at Dataiku (820.35s), Operationalizing AI Projects (982.73004s), Deploying AI Projects (2226.04s), Governance and Monitoring (2638.5251s), Testing GenAI Applications (3309.245s), Model Orchestration Options (3399.58s), CI/CD Best Practices (3433.65s), Final Questions Addressed (3491.1199s), Orchestration Levels Explained (3568.92s), Conclusion and Farewell (3602.96s)
Transcript for "End-to-End XOps With Dataiku": Okay. Dataiku. Welcome. Welcome. We have a couple people here already, and we are very thankful that you're joining us today for our AI operations framework webinar, why 46% of AI models fail and how to fix it. We are gonna jump in in just a moment. I always like to leave a couple minutes for people to join before we really hop into the nitty gritty. So in the meantime, please go ahead and post where you are watching from. I know we have people watching from all over the world, usually. France already won, that is international at at least from us. I am currently based in New Jersey. And, Chris, where are you currently based out of? Denver, Colorado. So we got a good spread. Yeah. We got a good spread of The US here. We have a couple more people sending greetings from Houston. We have Massachusetts, Raleigh, North Carolina. My old stomping grounds, I used to live in Raleigh for a little bit. We have Ecuador. We have London, Saudi Arabia, Lebanon, Germany, Stockholm, Sweden, Katy, Texas, India, Dallas, Texas, India as well. We have another Denver. So this is awesome. We have people joining from all over the world, which we do appreciate, and we hope that you guys all learn something from your different pockets of the world today. So we are about two after. We're gonna go ahead and get started. As we get started, please continue to leave where you're coming from as you join. Also, just to note, this will be recorded and a recording will be sent out to you after this webinar. Probably expect an email tomorrow, and you will also get a copy of the slides as well. So if you miss something, don't feel like you'll never hear it again because you will in that recording. So let's start with the hard truth. 46% of AI models fail to deliver on their promise. They fail to even make it to production to even start generating any sort of business value. But here's what most people get wrong about why this happens. It's not because the technology isn't ready or the algorithms aren't smart enough. The real problem is operationalization. Right? Putting those models into production. Most organizations are missing some sort of systematic foundation that's needed to get AI from a good idea to real business impact. So today, we're going beyond the surface level ex, explanations and we'll show you exactly why models fail and what separates AI leaders from those that are still looking to get started and most importantly, how to build the complete AI operations framework that gets your models to production fast and keeps them performing reliably. So let's go ahead and dive in. Here's today's agenda. We'll start off with Dataiku, you know, kind of what we do, but also understanding it from, the AI operation standpoint. We're gonna be understanding ops itself. We're gonna understand ops in Dataiku. We're gonna see a live demo, and then we'll have time for q and a towards the end. So when it comes to questions, please go ahead and leave your questions during, the session. We will be answering them in the chat, so you can go to the q and a section. It should be within your chat window. It should say q and a at the top right. Just go ahead and leave any of your questions there. We will answer them during the session. But also, if you want to save your questions for the very end for the ten minutes that we'll have in the q and a portion, that is totally fine as well. So let's get well acquainted before we hop into the actual information. My name is Chad Covin Kovan. I'm a senior technical specialist, a part of product marketing here at Dataiku, and I'll hand it over to Chris to introduce himself. Yeah. Hey, everybody. Principal solutions engineer here, which really means, expert in all things Dataiku and get to spend a lot of time with customers, helping them get value out of it. Yep. And we appreciate Chris joining us today. So let's go ahead and hop into, our, you know, all things ops. But before we do that, we wanna hear from you. Right? We want to understand some of your issues, some of your challenges, so we do have a poll to start things off. And I went ahead and opened it and we'll share it on the screen as well. So if you want to vote in this poll, go up to the poll section in your chat window and go ahead and vote. So what is your biggest AI operations challenge right now? Is it managing multiple tools? Is it monitoring model drift and performance degradation? Is it streamlining deployment pipelines to production? Is it balancing governance requirements with speed? Or is it coordinating ops across different AI systems like, models, agents, etcetera? So go ahead and leave a vote. We just want to understand you guys better and some of your challenges. And, you know, me personally, I love participation. So go ahead and leave the votes. We will go ahead and leave this up. Right now, you can see the real time scores. We see a lot of votes in terms of managing multiple tools. We see many, in terms of coordinating ops across different AI system types. And we even see a bit of model drift and performance degradation as well. So I'll leave this up. You guys can continue to vote, and we'll go ahead and start getting into the actual content. Alright. So now that we've gotten well acquainted, you guys are continuing to vote, which we do appreciate. Let's really break down where this operational breakdown happens. Right? So we know that statistic of 46% of models fail to make it to predict product, production. And also even 40% of models continue to degrade a year after they're put into production. So where does that breakdown happen? Well, most AI initiatives, they start small. Right? So teams build point solutions, they get quick wins, and maybe use whatever tools feel familiar to them. At this stage, about 88% of companies are diving into AI and things look promising. But here's where the cracks start showing. As success spreads to the department level, you need systems that actually scale. You're no longer talking about one model, you're talking about dozens. So for instance, companies like m and m Direct went from individual experiments to hundreds of models in production with Dataiku around while they were around this size and scale. So without proper operational foundation, teams start spending about 40% more time just getting models deployed instead of building better AI. And at the enterprise scale, this is where that 46% failure rate really kicks in. You're managing entire AI portfolios across data ops, ML ops, LLM ops, and now agent ops. And most organizations completely fall apart here because their operational foundation was never built to handle this level of complexity. So instead of innovation, really your best people are drowning in deployment overhead and fragmented tools. And that's exactly why Dataiku has the Universal AI Platform or we are known as the Universal AI Platform to unify all these operations to help you scale efficiently. So practically, what does efficient scaling look like? Let's walk through what any serious AI project actually involves. So every AI project starts the same way. You're pulling data from multiple sources and building the infrastructure to handle it. ETL processes, IoT hubs, retrieval systems, LLM repositories, this is essentially the foundation. Right? Then you need the compute layer, data lakes, warehouses, observability systems, knowledge bases, and access to large language models. So more most organizations stop here and they think that they're ready to build AI. But then reality hits and you need to actually build something. So last mile ETL for cleaning data, exploring data with exploration tools, data catalogs, ML platforms, and now AI agent builders. Each team picks their own tools and suddenly you have a ton of chaos. And that's before you even get to the operations piece and you have business intelligence, you have reverse ETL, you have ML observability, MLOps, governance, LLM ops, and every piece needs monitoring. Every process needs oversight and nothing is talking to each other. Right? So now you have these outcomes that have maybe a shaky foundation because you don't have, all of these processes actually talking and interacting with each other. Now multiply this across AWS, Azure, OpenAI, Anthropic, and whatever other providers your teams are using and you've got dozens of disconnected tools and platforms. So this is exactly why an orchestration layer like Dataiku is needed. Instead of managing dozens of fragmented tools, you get one platform that unifies everything from data to models to agents with governance built in from day one. And that's exactly how you scale efficiently. So now remember all of that fragmentation that we just talked about. It doesn't just create technical problems, but it creates human problems as well, and those human problems are what actually kill AI initiatives. So here's what we hear constantly from teams that we talk to trying to scale AI. Data engineers are burning out just trying to stitch systems together. Data scientists are spending more time on documentation than building better models. Operations teams are saying, you know, deployment is brittle and ML and AI engineers are drowning in compliance questions and slow workflows. Meanwhile, leaders can't get visibility into what is actually working. And every single one of these problems gets worse as you scale because when you have fragmented tools and disconnected processes, adding more projects doesn't just add more work, it adds exponential complexity. So now, how do we fix all those human problems that we just talked about? Dataiku's approach to AI operations comes down to three core principles. First, unify. Instead of forcing teams to work across fragmented fragmented tools, we cover the entire AI life cycle in one platform. No more air gaps between data prep and model building. No more handoffs between deployment and monitoring. Everything flows together. Second is operationalize. We remove the friction that's killing your teams with automated documentation, simplified deployment, and built in governance. And this is the stuff that used to take weeks and it now can happen in days to minutes. And the third is to repeat. It's not just about having one and done projects. It's about building a standardized approach that scales. When your second project takes the same amount of effort as your twentieth project, that's when you know you've got it exactly right. And here's the key, everyone, and here's the key. Everyone has a role. Right? So everyone is working in the same system. The business SMEs and analysts start projects. The data scientist then build the data scientist build models. Control managers ensure compliance, IT operations, deploy safety, and, AI engineers can monitor the performance. They can all have the same workflow in the same platform in the same visibility for everyone. So this is how you go from all of those frustrated quotes that we just saw to having teams actually enjoying building AI together. And I wanna be clear about something and that's AIOps is not just some buzzword that has popped up overnight here at Dataiku. We've been systematically building these capabilities for years because we saw this problem coming. So back in 2021, we started with the basics. Helping teams deploy and monitor more models across teams with the model evaluation store, automated drift analysis, risk qualifications, and model sign offs. Really the foundation of reliable MLOps. In 2022, we expanded confidence and control. Flow document generators so teams can stop spending weeks on documentation, model stress test, and feature stores. We were solving the governance problem before most teams realized they needed it. In 2023, it was all about adaptive multi platform ML ops. So enhancing deployment policies, support for external models, deploy anywhere capabilities. And because technology is continuing to grow and evolve, the best AI strategies work with your existing infrastructure and not against it. And last year, we moved from MLOps to really xOps. The idea of a full unified operational framework. So unified monitoring across all your AI types, data quality rules and dashboards, LLM evaluations. And because now you're not just managing individual machine learning models, you're managing AI agents, you're managing complex project portfolios, and entire application health across your business. And this has been a major focus for Dataiku over the past few years. Now let's understand how the ideas of unify, operationalize, and repeat fit into this. So let's start with unify. Today, data scientists build in one environment, MLOps deploys in another, governance happens in a third, every handoff is really a failure point. And Dataiku unifies this entire life cycle. Whether you're working with Databricks or deploying to AWS or Snowflake or Azure ML, it all happens in one connected system. And this isn't about replacing your infrastructure, it's about seamless workflows across what you already have. When unification works, teams solve business problems instead of fighting tools and that's what separates AI leaders from everyone else. Next, operationalize. So remember all of those quotes that we saw about teams spending weeks on documentation and brittle deployments, this is where we eliminate a lot of that friction. So Dataiku automates the operational overhead that's killing your teams. There's auto documentation from models and projects. There's open API integration so that you can easily understand what your APIs are looking to achieve and unified monitoring across all deployments. So when your ML engineer can push a model to production in minutes instead of days and weeks and monitoring becomes automatic instead of manual, that's when AI operations actually work at scale. And finally, there's repeat. This is where you go from that one off project to a standardized operation that can actually scale. So unified monitoring gives you a single view across all deployments whether they're in Dataiku or external platforms. There are test scenarios and project QA that let you verify projects and, verify that they are production ready with automated testing of your project workflows, Python libraries, and web apps. And this is integrated into your deployment pipeline. And there's Traces Explorer which lets you debug AI agents by seeing exactly what they're doing at every single step. So when your tenth AI project takes the same amount of effort as your first one and when testing and monitoring work the same way across all platforms and debugging of your agents becomes systematic, that's when you've built repeatable operations. And that's really the difference between experiments and having AI transform the way your organizations work. So now, let's actually see what this looks like. Right? We talked about why 46% of AI models fail and we've shown you the human cost of these fragmented operations. We've outlined Dataiku's approach of unify, operationalize, and repeat. Now it's time to see this in action. So I'll go ahead and hand it over to Chris, who's gonna walk us through exactly what this looks like in practice. Perfect. Alright. So we're gonna be talking and walking through a project today that has to do with this sort of one pane of glass concept for, I think, something we can all understand. Right? Suppliers, contracts, invoices, all this complicated data. And like Chad's saying, it's really the the before is that this is hard to do. Like, so many ingredients, so much tech, so many skill sets, so many handoffs. But I wanna show you how we can involve these different folks in these different technologies in a way that makes a lot of sense. And so the end result of this demo, kind of the reason we wanna operationalize it, is because we wanna give a front end to augment our operations SMEs. Right? So they can ask questions, kick off things in other systems, have one portal to get information and do things with invoices, contracts, suppliers. Now when you think about getting this to production, this is gonna involve a lot of people. It's not just your ops person who gets the project. Right? There's the SMEs and the data scientists who contributed to building the project. There's your AI engineers who are helping set up the monitoring, helping make sure this actually gets deployed and is tested. And we're gonna have folks like risk managers who actually understand the scope of the project and can do things like give sign off before it moves into production. And what I really want you to take away and see from this demonstration is that there's gonna be a wide range of folks, like I just said, but also a wide range of technology and process checks that are gonna be put in place, and those are all gonna feed back to a central monitoring area so that you can get one place to understand what's going on. So I'm gonna go ahead and share my screen now, and we'll get the show on the road. One second. Perfect. Okay. So we are going to start with the end result and talk about what we built, kind of take on the persona of our data scientist who's really work, you know, AI builder, who's really worked on this project in collaboration with an SME so you can understand what we're trying to operationalize because that's critical. So in this case, we have this one chat interface where our operations SMEs can come in and do things like say, hey. I wanna process the latest PDFs. Have an agent go off, process, extract key information, put that into a snowflake table that's ready for working. We can also ask questions about what just got loaded. You know, can you tell me if this particular order that got loaded will be delayed? Call a predictive model in the background, run it up against all the model the features that are needed for that model. Give us a score. Let us know. This one is, you know, likely to be delayed. We could also ask a whole bunch of other structured and unstructured questions that we need to get done. Could be questions about, you show me my invoices and contracts sorted by amount. Show me a table of them. Maybe I wanna investigate data call technologies a little more, find out more information about that contract, see that it's a lot longer than I expected in twenty four months. And, you know, do I have any other contracts with terms like that? And understand kind of the sources and why I know that contract was larger than twenty four months and was ten years. All kinds of things could be done with this front end interface. The whole point is to augment those operations SMEs. But, really, what goes into this front end interface is not necessarily simple. Right? Lots of ingredients to go into this. And those are all represented by this data I can project that my data scientists built. And, you know, as data scientist, I'm proud of this project. I have an area where I'm actually reading in invoices. I'm I'm processing them and getting information. I even create a knowledge bank that's used in that front end so we can ask rag style questions of those invoices and contracts. I'm merging that contract data with invoice data so I get one great structured data that's perfect checks the SQL and the agent that calls that and answers structured questions. I've even got a pipeline that's gonna help me grab data, load it into Snowflake, and create a predictive model about whether my shipments are likely to be delayed from a particular supplier. And all of those ingredients, right, all these different data sources, these different pipelines, these different models are all then integrated right into several agents. We have one agent that is actually doing the invoice extraction and sending the data to a table, and it's using a number of built in Dataiku tools to actually do that extraction and that writing. So I'm not having to code in this case. We have another agent that's actually doing the delay. You see it's pointed right at the predictive model. So it's referencing that, looking up the necessary data, referencing that model, and giving us the responses. And in fact, you know, that's just a taste of what you can do with agents in Dataiku. This is really focused on operationalization, but just to understand. We could have done a lot of other things, used a lot of other built in tools to do everything from, create Jira issues to, performing graph rag to talking to, ServiceNow or Salesforce, all kinds of things at our fingertip, and, of course, we could code to customize. Now as the data scientist, I built all these ingredients, and I have my front end and all these pieces. I'm pretty proud of this. And I actually did go a little farther than I tried to do a little bit of operationalization style work myself. I made sure that I have all these logging and monitoring datasets piping directly from that chat front end. So we know things like the conversation history. We know messages that are coming and feedback from my users. We also built in some other checks on our our modeling area because I'm I'm pretty familiar with modeling. I felt confident that, you know, I could build some data quality checks on my data feeding my model. You know, I've chatted with some of my SME friends, and I know that we wanted to build some checks on the particular delay column that's so important for our train data, make sure that it's within typical boundaries, maybe before we decide to retrain a model. If the delay is looking really strange, maybe we need to investigate before we just retrain and put a new model into prod. And I even decided to really work with my SMEs to to ask a question and guilt some extra checks on my model. So here's my traditional ML model, and I actually went ahead and said, you know, hey. I noticed I'll zoom in a little bit here. I noticed that we have a supplier called DriveWorks and seems to be really point that they cause a lot of delays all the time. You know, do we expect this to continue? My SME said yes. So we actually work together to go into the model, this logistic regression. And right within Dataiku, we have this area called metrics and assertions, and I can go ahead and build that custom rule in. I could say, you know, you know what? Every time we get DriveWorks, we really expect a decent chunk to be late. And if that doesn't happen, I wanna have that business rule alert totally separate from any kind of performance metrics so we can make changes and be aware. So pretty cool. I was able to build a lot of different things, add a little bit of operationalization. Right? Add add some data quality checks, add a little kind of business rule on my model, set up some monitoring, logging. So I'm feeling pretty good. I'm gonna go ahead and tag in my AI engineer. I'm a little bit more of a junior data scientist. I don't really have the permissions, and I'm also not comfortable putting this right into pre prod than getting into production. I'm gonna tag in my AI engineer colleague who's gonna take a look and see what else needs to be done for this really to be up to snuff. So now we'll put on that AI engineer hat, and we'll come and take a look at this project. And one of the nice things is, as the AI engineer, it is a lot easier for me to understand what's going on. There's things like comments, you know, here's my invoice knowledge bank. I've noticed that my data scientist colleague actually built a Wiki so I can understand what's going on, which has links to automated documentation for the project. And I can even take a look at things like tags and say, you know what? Let's see. Oh, okay. Data quality tag. I see we put a data quality check on what's feeding the model. That's a great sign. That's something I would definitely want to be involved. So you can see this this framework is a really nice one for helping me understand what's going on, but we're gonna have to look a little bit deeper to see what we have. And the first thing that stands out to me before I even worry about complicated automation or monitoring is this knowledge bank. And I wanna understand what sorts of LLMs my data scientist colleague is using and whether they have the proper checks in place. So let's hop into this knowledge bank and take a look at what's going on. First, I can notice they're using GPT four o in this connection, and that actually makes me feel really good because the connections in Dataiku created by our administrators actually have a lot of checks and guardrails already built in. So if I look at the configuration for this connection, how Dataiku is connecting to OpenAI, for example, I can actually see we already have a couple guardrails in place. Things like PII detection, toxicity data detection, and we can add quite a few more if we're administrators and have permission. But I like that we're using that built in connection. Gives me confidence. We already have guardrails in place. But what if we wanna peel the onion a little bit and kinda do some some other live checks on how well our knowledge bank is serving up its documents? And in that case, I'm gonna go ahead and turn on our faithfulness guardrail. Basically saying, when someone asks a question, if the generated answer doesn't match the document very well, like, if the LLM is using the context and doesn't look great, we're just gonna go ahead and overwrite that live and say ask another question. And that'll show up in our logging for us to analyze. So love this live check, really easy to turn on, doesn't require any code, make this chat a little bit safer as these operations SMEs are using it. But I think anything we need next is gonna take a little bit more work because the biggest thing I'm seeing here is we're not actually tracking a lot of metrics. Like, we have this logging and monitoring, but I don't see anywhere that we're actually tracking things over time programmatically, setting ourselves up to be able to get alerts and kick off automated actions. So I'm gonna go ahead and come over here, and I'm going to show you what I've added, these AI engineer additions, to this project. And so let's start from the bottom. First, I took the conversation history. So this is information about questions actually asked, how they're answered by the LLM, and then actually live feedback in the feedback message given by people who are using the app. You know, positive, found the right contract. And in Dataiku, it's very easy to build checks on this, where I can say, I wanna build my own quality checks, but on this and I wanna make sure that if the amount of negative feedback I'm getting falls outside the range I'm used to seeing, you know, just a couple a day, I wanna make sure we get an alert so we can do things like tune our prompts, so we can do things like maybe incorporate more golden answers into our few shot prompts, maybe even fine tune the model if it really calls for it. Okay. So we're getting that feedback in. I like that, but not everyone's gonna give feedback. And I really need another more programmatic way to get a handle on my agentic gen AI pipelines. And in this case, I'm gonna take the much broader messages logging dataset that my data science picked out, and these are things that are easily generated by Dataiku just chosen as drop downs inside of our chat front ends. And I'm gonna use something out of the box Dataiku has called an evaluate LLM recipe. So I'm going to use this to point at the things that are flowing through my chats, the questions, the answers, the context, what the agents are finding and using, and we're gonna use that to generate a bunch of metrics. And it can be as simple as saying my input questions or queries, my output columns or answers. The context is where I can see everything the agents are doing or the rag documents that are thrown up and used. And then I can use either a bunch of built in or even custom metrics. And in this case, I'm gonna use some answer relevancy and Bateson's metrics, and I'm gonna tell those metrics to be computed by another LLM. In this case, I'm gonna leave it with four o. The way I want you to think about these is not like these are the perfect metrics and that you don't need human in the loop feedback, but when I set this up, I now get all of these metrics recorded in a Dataiku LLM evaluation store, this green diamond here. And that LLM evaluation store is gonna let me record, understand, and check these metrics. So if we hop in, you can see how over time, my answer, relevancy, and faithfulness both look like they're staying about the same. And if I wanna get a little more information about what's going on, I can actually go into the row by row analysis, see all of those messages. Maybe I'll come in here and sort by answer relevancy to look for my least relevant answers. And I can see, for example, someone asked, can you find the Allied Sports Entertainment contract? My agent responded, I don't have details on that. And you can see that we tried to look in a particular KB and couldn't find it. So this is a great example of helping me understand if something's in production, not yet, what I might need to do to go answer and understand. Do we need to inject more documents into the knowledge base? Do we need better, instructions in the app to let people know what contracts are available before they ask questions? So besides just looking at these metrics and diving into the row by row, I also get the ability to set actual checks to say, hey. If relevancy falls below a certain threshold, I want you to give me a failure. Right now, we get a green, but in the future, we could get a warning or an error, and you're gonna see how we can tie that into automation that actually makes things happen. Right now as the AI engineer, I'm more focused on just setting up the ingredients. I wanna make sure all of the checks are available, then we'll take those ingredients and make sure they're automated in the right ways. Okay. I'm feeling better. I now have feedback checks. I have my more programmatic checks on my agentic work, but I can't forget about that traditional ML model. We all know that, maybe traditional ML models are more deterministic than GenAI, but that doesn't mean they're not going to degrade as conditions change. So this is one of the things I really like as an AI engineer about Dataiku, is that I can take my traditional ML model, have my ground truth data pipeline, and I can use another built in evaluate. But instead of evaluate LLM, I can actually use my evaluate regular ML model. I can still pick my metrics. I can still decide what I want to do, and I get a similar evaluation store. So I have my evaluation on the messages from my agentic LLM work, and I have the evaluation store on my traditional ML work. And if I hop in, the interface is different but similar. Right? I'm seeing things like how accuracy and precision and recall are moving over time. I'm also seeing how my data and my predictions are drifting over time, which are useful even when I don't have ground truth. Similarly to that LLM eval, I can investigate more deeply. I can actually see what's going into my input data drift as I compare my train time data against my current ground truth data feeding in. I can even sort by my most drifted features and kind of see the difference. I can see in supplier name. You know, in my current data, there's a lot less auto supply than there used to be in the train data. Might wanna investigate why. I can understand prediction drift. You know, it used to have 27% predicted, 27%. Now I got 32% predicted false, and I can understand my performance drift as well. And what's key here is, again, this is how once I find an issue, once I get an alert on drift, that I can do some investigation if I want to, right, if I need to understand before automated things are kicked off. Just like with the l m e l m eval, I can also set those thresholds so that I get automated checks on things like AUC, on the prediction drift, on data drift, whatever I want to be alerted, depending upon whether I have ground truth or not and what I'm most interested in in this pipeline. And the great thing is that I didn't have to build all these from scratch. I actually went to some best practice template data, projects, brought these things in, made some changes, looked at the right field, and I'm feeling a lot better about the monitoring I have set up. So got that KB, that knowledge bank updated. Got a bunch of additions on how we're monitoring things, but in my head, the really big thing that's missing is how we bring all this together. Before I put this into pre prod and run all my checks, I need to make sure that I can automate these actions, get alerts, do things like kickoff retrains, and also make sure that I have a bunch of good checks so that when this moves to pre prod, we can know for sure with the testing framework that things look good technically. So how do we get to that piece of the puzzle? Well, in Dataiku, there's a couple of things that matter. And the first thing that matters is that I can set up these automations through Dataiku scenarios. So the same scenarios you can use to automate a data pipeline, you can use to automate these sorts of drift checks and understanding. So in scenarios, you're gonna see three. I've got drift retrain from my traditional model. I've got LLM alerts that I threw in here to understand when things in that programmatic, option are getting off beat. And then I've got some test scenarios, which we'll talk about last. That's some pretty interesting stuff. And, really, you know, when I think about my collaboration with the data scientist, if that data scientist was a little less junior, I would have hoped maybe they put some of these things in. But these test scenarios are a great example of where I'd really expect the AI engineer to come into play. But let's understand scenarios if you're not familiar. So for scenarios, I can tell them when to execute based on time, based on things changing. I can have different reporters and alerts based on, you know, when a retrain occurred, for example. And I can set exactly how these things happen. I can say, orchestrate all those ingredients we just set up. For example, build the evaluation store on top of our delay model that records all of those metrics like AUC. Run the checks we built. See if it's below the threshold. If and only if it's below the threshold, retrain the model. And then run a new check on the model itself, make sure that newly retrained model is looking good. And what's nice about scenarios is that when these things run-in production, you're gonna see it's very easy to look at visual logs to understand exactly what happened. We built the evaluation store. We had warnings. We verified the rules or checks. We had a red x. That means we retrain the model. The new model looks good. We can do something similar for l l at eval where instead of kicking off a retrain, because we don't really have a fine tune model or something like that in this case, the main thing I want is that when things fail, I wanna get an alert that's letting me know faithfulness and relevancy are declining. We need to go understand the feedback. We need to go understand how we can fix this pipeline. Okay. Cool. So we're we're now at a point where we feel good about orchestrating all these different pieces. But what I don't know is when I take this project, which we're gonna do in a second, when I feel good about pushing it into a QA test environment, pre prod, whatever you like to call it, I wanna make sure the proper CICD checks are run, that I can make sure the right ingredients are there, that things that meet our standards as an organization from a technical perspective. And in this case, in Dataiku, we have this area called Vibras. It's this great asset as an AI engineer where I can simply point this at my approved Git repo and pull in a bunch of checks, pytest style checks for those who are familiar to say things like, hey. If we have a knowledge bank in the day in the the project, do we have that guardrail enabled, what you saw me turned on? Because if not, we wanna know. We wanna have a failure. If we have LLMs involved, do you actually have that LLM evaluate recipe in that model store? Are you tracking the things you should? And if you're tracking the things you should, do you meet the thresholds that are necessary for this to work? Now you can have all kinds of tests here. These are just three examples that I think are really useful for my organization in this project. But because this is hooked up to Git, you get one place to update the tests. These are then automatically going to run when this hits pre prod because they are turned on within this test scenario. And you can see the little purple bug, you can see that that's designated as a scenario that should run when we hit pre prod. So we have that all kind of nicely bundled up for you, and you're gonna see these test scenarios, even create a small little report that we can verify and understand, make sure things look the way they should. Okay. So as this AI engineer, I have done quite a bit. Oh, yeah. The data scientist felt pretty good about where they were, and I salute them for the tagging, for the documentation, for some of the data quality checks they built. But we added quite a bit. Right? We added quite a bit of checks and monitoring, both traditional ML and GenAI, and we also, add some other pieces. One thing I did neglect to say, we also did make sure we're actually having that Trace Explorer web app that Chad mentioned ready to go. This is an out of the box web app that doesn't require code that comes with Dataiku so that when this goes in production, me as the AI engineer, I can actually come in and see things like, hey. Remember that question you saw in the in the chat? Can you tell me if this particular order, CA8, is delayed? That now shows up in the logging datasets, and I can explore it with my trace explorer. You can actually see things like agents, guard services, tools, LLMs all working together here. And if you wanna see how I might explore this, let's go to the timeline view. Where I can actually understand, I'll click my first completion, and I can see, hey. You know what? The user's asking if a specific order will be delayed, which aligns with the functionality of our delay agent. Great. That makes sense. It seems like we're calling the right agent. If I wanna go a little more deeply, I can actually see the agent call, and then I can actually see the tool calls if I kinda highlight tools here. I can see here's my tool. It was my lookup supplier tool. This is my dataset row lookup to get the information the model needs to do the scoring. And if I look a little farther down, I'll zoom out a tiny bit so we can see, we can actually see another tool. This is that classical prediction model. This is that built in tool calling the model to say yes. The response says that's delayed, and we can actually see the response up here. The order number with this is predicted to be delayed. So this view is essential when you think about how do we diagnose and understand when we get some of those alerts, when things aren't going wrong, when we're getting bad feedback from our end users, how can we understand and explore what's actually going on magenta? Alright. This project is feeling pretty good to me as the engineer. Let's now bundle this whole thing up. That's Dataiku's way of taking all of the pieces of this, the pointers at the data connections, the evaluation stores, our traditional model, our knowledge bank. Let's get all that together and let's move that into another environment. And we do that by publishing it to what Dataiku calls the deployer here on the top right. Think of the deployer as Dataiku's sort of artifactory where we can stage assets before they move through our different environments. Okay. Let's go look at the deployer. And here we are zoomed in on that contracts and suppliers project we're talking about. And you can see here I have a version in pre prod that we just pushed. And here's where I, as the AI engineer, can get a lot of feedback on what happened when I push this into pre prod. I can see things like the fact that when we pushed it in, it imported the bundle, it loaded it into the new environment, and then it ran. That hook automatically ran those test scenarios we talked about. It says one scenario is ran. Great. That's running all of my checks. We got a green. That makes me feel good. And some results got pushed to govern. We're gonna talk about that in a second when we take a step away from the AI engineer and get into that risk manager persona. But for now, as the AI engineer, this deployer is really my one stop shop to understand what's happening. I can see my test scenarios. I can actually see that on the latest runs, all the pieces ran. We got the green check marks. It's feeling good. We've got all the right pieces. The CSC checks are passing. We can see that our drift retrain is set up to auto trigger, but it hasn't run yet, which makes sense. We're only telling it to run once a week, so we haven't run that in pre prod. I can also see that my main app that matters, the back ends up, my web app's running, it's ready to receive requests, things look great. So overall, added a bunch of things into the project to get it ready. These copyable, reusable, searchable things that make it really easy to put things into production. I put it into pre prod. My check checks have passed. But what I wanna emphasize is that tech checks aren't the end of the puzzle for these types of projects. This is the Gen AI project. It's gonna be looking at a bunch of data, probably needs to be looked at in a more sensitive way. And in that case, we can actually head into Dataiku, a central watchtower to see overall of our projects and put on that risk manager persona. So here I'm on Dataiku, and I can see the sort of governance twins of all of my projects that I have permission to see as that risk manager. And in this case, I can see my contracts and suppliers project. I could see it's been ranked as medium high value, medium low risk. Give me some context. And I can get a lot of information about the project. But, you know, I don't need to go into the project. I'm a risk manager. I'm more interested in what business initiative does it roll up to, who is the sponsor, what's it about, you know, what does it involve, in this case, ML analytics and GenAI. I can see all of the things that are used, the agents we mentioned, the model that's mentioned. And if I really wanna dive in, I can actually go to the particular thing we just published, the latest version. And as that risk manager, I can see that it's currently in review. Now that's gonna mean that that project in pre prod can't get to production. It's actually gated because it's been determined with a risky project that I have to be able to give this sign off. And in fact, that review can involve lots of folks. It can have IT and operation stakeholders, risk and compliance stakeholders like me, stakeholders from the business, and everything from the steps this follows, this project, to who does the feedback can be totally customized to fit your organization. Now we're gonna do a little simulation. We're gonna say we're gonna request the feedback, and we're gonna skip right to the final approval. We're gonna assume that me as the risk manager, I have that final approval. I'm going to say yes. And in fact, before I hit that final review and approve, you can actually see something pretty interesting, which is we have a test report here in supporting documents. And this test report, if we download it, is actually the result of what ran. We can see that the web app works, LME belts present, the threshold mostly there, and the KB guardrails are present. So what I really want you to know here is this is the integration of things like those check checks and these people process checks. So then maybe someone from IT and operations could make sure they view that and understand and approve even if they never go into the Dataiku Deployer. But let's go ahead and let's say we've met the standards. We feel good about this project. I'm gonna give this an approval and say meets tech and risk standards, and this now talks back to our Dataiku Deployer. It can even do things like kick off an automatic move into production. Depends on how automated you see your ops process in your organization. But if I go back to the Deployer, we can now actually see that contract's in production, and I can actually get more information here. But what I wanna show you is that this is where you look at one project, but Dataiku also pumps all of the information we just talked about, the governance information, the model information, the data quality information back to one monitoring area within this deployer. So now I can get one place to see all of the projects, all of the API endpoints that I'm responsible for. I'm back in this AI engineer persona where I need to keep a take a look at all these different projects. And if I want to zero in on, let's say, my contracts and supplier, let's go ahead and filter to it. I can see my one version in pre prod and my version in prod. And if I click, this has all of the breadcrumbs I need to understand what's going on. I can see things like the web app starting just like we saw before, but I can also see that my drift free train scenario did kick off in prod and it looks great. I can also see it when there are issues like things with my model status. I can see my data quality that was looking at my feedback dataset and my train dataset feeding my model. I can even see with links directly back to the governance artifact that we just saw where we got the feedback and review if we wanna check off, you know, how did it get sign off. So this monitoring becomes the one single pane of glass for that AI engineer to understand all of their projects and API endpoints even if some of those endpoints are deployed not in Dataiku. Just something like Azure ML or Databricks or AWS, like Chad was mentioning in the beginning. So I know we've hopped around a lot today and shown a lot of technical and people process pieces, but I want let me just kinda wrap up and tell say what we talked about. You know, overall, we started with this junior data scientist who worked with an SME to build a pretty complex project involving traditional ML models, agents, vector stores, front ends, all kinds of pieces of the puzzle. We then kind of tagged in our AI engineer who added some checks on our knowledge banks, added in a bunch of monitoring, all with built in reusable Dataiku evaluation stores, data quality checks. We then automated every piece of the puzzle through scenarios, both scenarios that are to do things like retrain our model and alert us to LLMs, but also more operational CICD checks that we could pull in from Git to run. We bundled our project up, moved it into our deployer, were able to seamlessly move that project to pre prod, have the checks run and verify it. We then went into Dataiku with our risk manager, gave sign off, and we're even able to review those risk report outputs. Finally, we got that model into production, and we're able to see everything that happened back in Dataiku. So I really hope this demonstration has made what Chad was talking about a little more real and shows you how you can have a very systematic repeatable process for taking really great work like that supply chain invoice front end and getting it all the way into production with all the monitoring you need. So I think now for the last nine, almost on time, we can go ahead and open it up to questions. I know Chad's been answering quite a few in the chat, but, Chad, is there anything you wanna surface up to me? Yeah. So, first and foremost, thank you so much, Chris. Also, what I do want to do as you guys are going ahead and leaving your last little bit of questions, I have opened up one more poll so you can go over to the polls tab and let us know after seeing today's demo which unified ops capabilities would you, would have the biggest impact on your AI initiatives. So, one of which being automatic drift detection across the pipeline, faster deployment, built in governance, end to end monitoring. There are many to choose from there, so I recommend you head over to the polls tab and go ahead and vote. Let's go ahead and answer some of these questions. One that I think you answered within the demo, Chris, but, you can go ahead and and add some light to this. How do you evaluate the performance of this RAG pipeline? Can you be certain that it always, is going to retrieve accurate information? It's a good question. And the first answer is can you be certain? You definitely cannot be certain. In my opinion, with where we're at, you can get really close, but there's going to be, issues even with, you know, changing different model types and things happening over time. So that's why I usually do wanna take a two pronged approach. One approach is a human in the loop feedback system. And in this case, I was mostly monitoring the feedback, but you definitely can work that into golden answers to help you evaluate your pipeline more programmatically. And then you can also use Dataiku's built in LME about to set up those monitoring pipelines to do more LMS to judge metrics. But a %, all LLM pipelines, in my opinion, and agentic pipelines need to be closely monitored with human loop feedback because at this point, I do not believe we can be 100% sure if things are working exactly as they should. Yep. Very, very true. Alright. There is another question. Is the underlying compute engine in Dataiku completely managed, or can we configure and build a compute engine ourselves? That's a that's a really good question. And Dataiku is something we talked a little bit about today, but not a ton because it wasn't a huge focus, was that Dataiku is 100% primarily an orchestration engine. So we want to use your best in class tech. I don't want to suck everything in into the Dataiku engine. So at various parts, you know, when how does my rag pipeline get rebuilt? Where does my web app run? How do I set the retrain in my model? You have many different options from containerized execution on Kubernetes to pushing work down into a source system like like a Databricks or a Snowflake, and a number of other options. So and those can all be very, maybe for the AI engineer persona, much more open in what they can choose. Maybe for that SME data scientist, a little more limited, all very permission backed. So, absolutely, you can have a lot of flexibility in how things are computed and orchestrated. Yep. Totally. Totally. Alright. So next question that I see here, how many languages does this support, and what is the pricing? Well, answer the second part. You're you're probably gonna have to talk to our sales team about that, which we we definitely recommend. They're they're a great group of people. In terms of how many languages does this support, and I would love it, Samad, if you could add some context to this. I assume you're talking about, GenAI capabilities. If if that's the case, many languages are supported. Right? So really, it's dependent on what LLM you're looking to use, from which connection you're looking to use. So if you're utilizing, you know, something from OpenAI or Azure OpenAI, then you definitely could do, things in Arabic and English and in many, different languages, French, Spanish, you know, so on and so forth. If you're using maybe a lesser known LLM that's maybe a bit more specialized and a bit more, local maybe from Hugging Face. It's not guaranteed that it'll work in the languages that you necessarily want, but, if you're using more of the the best in class LLMs from your connections, you can definitely utilize many different languages. And in terms of Arabic and English, yes, definitely. If we're utilizing, GenAI is what you're talking about. I will just add probably not what the question was asking, but I showed pretty heavily visual pieces today to show you Dataiku's built in things. But you probably noticed in those, CICD checks, when you think about what you can use in Dataiku from a coding perspective, Python, SQL, r, Spark SQL, PySpark. And then for our web apps, you're talking about, using other languages as well for things like even if you wanna code like a React app. So very flexible if you do wanna bring an existing code as well. Very true. Yep. And we hit on both languages there. Right? So Yeah. Any other questions? We have about four minutes left in this session. Also, like I said, if you have not voted on the poll, definitely recommend. We'd love to hear from you all. And, let's see. Oh, did not mean to share my screen. I meant to go back to the questions. Is there a possibility to test it? Yes. So I would say the easiest way, if we're talking about languages and we're talking about utilizing GenAI, Chris didn't really show this in in this demo because it wasn't really the focus, but, you could definitely easily test it within prompt studios and Dataiku. And prompt studios essentially is a sandbox where you can go ahead and test your different prompts and your different interactions with all things generative AI. So if you wanna test, how an agent is working, or just how an LLM is interacting with, a RAG knowledge bank, you could definitely, do that within Prop Studios for easy iteration and testing there. And if you do if you do wanna get your hands on Dataiku, we do have I just put a link in the chat, a fourteen day free trial for our cloud platform. And when it comes to because I think this applies to Jasper on your question around test in a restricted organization, I think something that's key when we talk about orchestration, we're also just orchestrating models. So if you're not allowed to use OpenAI in your organization, we're not gonna be able to connect to OpenAI in your organization, but we do also, to Chad's point, allow you to point at and deploy open source, hugging based models. So I have some corporations who can't use OpenAI, but they can use, let's say, LAMA three, and we can use that. So the the answer is a little nuance, but there are definitely ways to make it work. Totally. Another question is any best practices for CICD with GitHub actions? That is a really good point. And I will put in, it's not the exact thing, but I'll put a link into the chat where part of the cool thing is I showed you the very Dataiku view of the world. Here's that link. But everything that I clicked and did and then I kicked off via Dataiku scenarios can 100% be called via API externally. So if you want to orchestrate with GitHub actions, an example best practice would be you can still take the bundle in Dataiku like I did, put it in the deployer like I did, but then you can have your GitHub actions do everything else. They can kick off test scenarios. They can run their own checks. They can move things between environments. So, I usually recommend if see if Dataiku fits what you need by itself first, but everything can be externally orchestrated. So I put a link about how Azure DevOps can do it, but very similar for GitHub actions. Totally. And, last question, we're, well, you guys can continue to ask questions. We do have about two minutes. So if we have some last minute questions, please fire them in. We have how do you get in touch with us? So I did just send a link. I'll also put it in the chat in case anybody would like to get in contact with us. You can go ahead, go to that link, leave your information, and our team will get in contact with you all. Last question. Any Infra as a code tool associated with Dataiku? So that's a complicated answer. If we do two things I would say quickly though. One is Dataiku can deploy a SaaS and Dataiku can deploy in your tenant. If we deploy in your tenant, we have built in templates for things like, Terraform so that we can deploy in your environment that you can, of course, use your own in bridge code if you want to because everything is available via API, but we have good examples. So I would say, is there anything that's associated? Not necessarily, except what we bundle, but it's very open to use whatever your IT organization wants to use for Ambrose Code. Mhmm. And we have thirty seconds. We will answer this last question. Is this orchestration at a platform level while something like LandGraph is orchestrated at a framework level? I I think that's a reasonable way to put it. It depends how you define platform and, other options. But, yes, just like, you know, we integrate with MLflow to do ML orchestration at the framework level, and we integrate with LandGraph to do orchestration at the framework level. But then we kinda give you the abstraction level up to tie it all together. So I think that's a that's a pretty good summary, Vignesh. Yep. Yeah. I think so too. So, thank you guys all for joining us, for this edition of Dataiku Live. We appreciate it. You will receive the recording, within this week. So thank you guys once again. And that's it from from Chris and I. Thank you so much, and, we will see you next time. Bye.