Video: Beyond the Headlines: Did MIT Really Say AI Agents Don’t Work? | Duration: 3616s | Summary: Beyond the Headlines: Did MIT Really Say AI Agents Don’t Work? | Chapters: Welcome and Introductions (5.92s), Study Reactions Analyzed (182.95s), ROI of AI (286.6s), Early AI Adoption (441.33502s), AI Implementation Challenges (618.035s), Custom vs Generic (787.365s), Fine-tuning vs. RAG (1010.41003s), AI Customization Strategies (1274.5349s), Scaling AI Implementation (1624.355s), Shadow AI Economy (1828.995s), Democratizing AI Innovation (2084.93s), AI Technology Decisions (2947.4s), Build vs Buy Decisions (3060.8901s), Q&A and Creativity (3242.205s), Feedback and Improvement (3378.435s), AI Governance Importance (3471.305s)
Transcript for "Beyond the Headlines: Did MIT Really Say AI Agents Don’t Work?": Alright. Welcome, everybody. Welcome to this joint Dataiku webinar. Today's topic, behind the headlines. Did MIT really say that AI agents don't work? You know, that that dog kinda cocking its head like, really? That was that was my reaction when when I first heard about that. Welcome, everyone. So we're we're gonna get kicked off, pretty quickly here. Don't hesitate to go and put some, messages in the chat. Let us know where you're, you're connecting from. Typically, we've got people from all over the world, so just head on over to the chat and and just drop it in there. It's a little reminiscent of the old, chat room days of everybody introducing themselves with their with their location. We don't need the other demographic details. That's that's, you know, happy to be, to be private. But, yeah, Arty, that's great. We've got Texas, Chicago, Spain, Berlin. I'm joining us from Stockholm, which is just a wonderful city and incredible weather today, which, which I'm very happy about. Mariam, I think are you in Paris today? I'm in Paris today, but, actually, the weather is much worse in Paris than it was in London this morning. So I'm very disappointed. Yeah. So I I flew out of Paris to come up to Stockholm this morning, and, I think I got the the the better, the better version of that deal. Yeah. This is great. We it looks like we've got people from literally, all over. Great representation from The US, from Europe. And I think that makes sense. Right? Because the, the topic today, that, that MIT study, certainly generated a lot of headlines around the world and certainly raised a lot of questions. But before we get into into that study, let's just do some quick introductions of ourselves. You know who, who it is, speaking up here. So that's me up on the top. I'm Kurt Walt. That's a slightly AI ified version of me. I'm not sure if I like it. You know, you can make these so easily now that you just test different versions. It's close. Right? I'm head of AI strategy at, at Dataiku. If you're not familiar with Dataiku, we are the universal AI platform, the single place you can go to build your analytics, models, and agents in a fully governed multi cloud environment. That's Dataiku. Miriam, so happy to have you here with us, with me. Would you like to introduce yourself? Sure. I'm Miriam. That photo was actually a real life photo, which are rare nowadays. Maybe had an Instagram filter on it, but that's it. I'm Maryam. I'm the cofounder and CEO of Double Word. We are a inference company, and specifically what we do is we help companies deploy language models in their own private environment. So either on prem or on VPC, and we work with Dataiku as a plug in within their own mesh. And I'm delighted to be here, Kev. Yeah. Well, I'm delighted to to be doing this webinar with you, because I think that, you know, this study, especially the reaction to it, really benefits from a little bit of not debunking, but just kind of explanation. So having your perspective on this, especially with your, you know, deeply technical background, I think that that's gonna be a big help for us today. So before we actually start with the study, I feel like we have to start with the reactions to the study because I think before anyone read the study, they saw the headlines. And the headlines were kind of dire. Right? It's like ninety five percent failure rate of generative AI. I feel like that is what most people, understood about this. And even right? You know, it had an impact in the market. I don't know. Mary, what was, what was your reaction when you first started hearing about this? I know I learned about it through, like, the business press. How did how did you hear about this study the first time? I think I heard about it on LinkedIn by some LinkedIn influencer saying this is the end of the world and AI is massively overhyped and it's a bubble. And in preparation for this webinar and when we're discussing it, I read a lot more into it. And I was like, actually, this is not that remarkable. And so, potentially, the outcome of this webinar will be that everyone was realizing that the study isn't that remarkable and actually just has inside it a lot of good lessons. So, yeah, the press really overhyped for this one, I think. Yeah. I think the, you know, obviously, everyone's looking for a story. Everyone's looking for a narrative. So I get it that, you know, things are gonna get hyped up and especially when, you know, there are some what may be on the surface surprising numbers that's gonna get promoted. But, yeah, I think that, through the course of this webinar, we're going to cut through some of that hype, some of that overreaction to to those numbers, and really dig into what the the study not only said, but really recommended for for companies as well. So, yeah, let's, let's dive in. So first things first. Right? That's the study. Like, that's the actual cover to it. You can, find it online. It's it's actually I I was having trouble to, you know, actually finding it, recently. I think they might be updating the study, but it's it's from MIT. MIT has this research initiative called, NANDA, which is actually like a really profound and, deep, initiative looking at how agents are going to connect, with one another in the future, like, in the GENTIQ Internet. It's super interesting. But part of that work, as part of that work, they ran this study, and they interviewed a bunch of people. There's the methodology page on the, just inside of it. They surveyed, you know, more than a 100 people, talked to something like 300 people, and summarized their results. And, Maryam, I see you just dropped the, the link into the, the chat. Thank you for that. I always have trouble finding it. I don't know I should just, I should just, bookmark that or something. Right? But yeah. So, let's just dive into it. I think where we need to start on this is that headline number. Right? So that headline number was 95% of organizations are getting zero return. And so we have to qualify that. Right? This was, going to be, or this was just in the executive summary. Just page one, you open it up. And if you only get the that far and if you're looking for a big number that you can put in the headline, this is, this is what you saw. So, Maryam, when when you saw that number itself, right, what was your reaction to to that ninety five percent so called failure rate? Yeah. It's interesting. I think I want to ground this discussion by reminding everybody that most people were not thinking about agents even two years ago. They weren't thinking about GenAI really until about three years ago. We were, but we were coming from, like, you know, very deep tech backgrounds of people who are at Dataiku. But we are really, really, really early when it comes to, like, the adoption of a new technology. And if you were to say to me something like, let's even take this number as true, that within the first two years, you know, you would have 5% of projects working really delivering genuine measurable ROI, then I actually think that's probably in line with potentially what you might expect, given we're at a stage where people are doing a lot of experimentation, figuring out what's working, what's not working. And, actually, you would expect this kind of slow gradual start of experimentation, and then for people to figure out their killer apps and figure out where they can really get ROI and then explode. So I just kinda wanna remind everyone that we are really, really early into this. And is it that surprising that most AI projects at this stage would fail while we're in that learning phase? Potentially and actually, probably not. And so that's, I guess, the the first thing. I think people often forget how early we are in need to I don't know if you also find that you forget that as well. Yeah. Totally. I what I what I've been doing recently, like, frequently, is thinking back to what is the equivalent moment to where we are now with GenAI and especially, you know, Agintiq AI in the enterprise and comparing that back to, you know, different periods as machine learning was becoming more commonplace outside of just pure tech companies. You know, so if you roll back to 2010, 2011, 2012, you know, that's when, of course, you know, you had a lot of big tech companies, the Googles and the Amazons, really perfecting a lot of the, you know, kind of the core capabilities, the core practice of, machine learning in an enterprise setting. They were building the first recommendation engines that were highly efficient, highly scalable, and really good at recommending products. Right? This was this was new back then. And so, you know, after that, a couple years after that, as you're getting into, you know, 2012, 2014, 2015, that's when you have things like scikit learn, which, which for those those people, who were not doing data science back then, you know, that was an open source or still is an open source, package of machine learning algorithms to do predictions, to do classification work. And it became available, and a lot of people started picking it up. And a lot of, you know, these nascent data science groups within enterprises were looking for ways to apply it. But, really, there was a a a lot of questioning around, you know, how do we actually do this? What are the use cases and what are the best practices? And just as, you know, in that period, we were identifying use cases, developing best practices. I think that's where we are now. Now what came after that, that period for machine learning? Well, the typical use cases became really clear. Right? We, we we ended up with, you know, this really clear idea of, you know, if you're not doing, omnichannel optimization for your marketing, you know, you you really should be. Right? If you're not doing a next best offer, you should be. If you're not doing customer lifetime value, you should be. And at the same time, we also then over the period of, let's say, 2015, 2016, 2017, also then perfected a lot of the capabilities for, for doing this and the the best practices for doing it in the, in the enterprise. That's where you have practices like machine learning operations coming up. And so I think for me, exactly like you, Maryam, when I read that ninety five percent, of the pilots are failing at this stage, my reaction was like, yeah. That sounds about right and sounds pretty normal for where we are in the enterprise adoption of this technology. Yeah. I guess, like, one of the things is, like, pilots are pilots for a reason. You know, we're trying out a lot of things. We're seeing what's working and what isn't. And so I don't think it is that surprising, to be honest, even if we take this as red. But I would also highlight that the way that they talked about getting zero return was actually fairly strict. They talked about it in terms of getting, like, very measurable, like, p and l drivers, which is, like, definitely a way of, like, measuring ROI. But you can think of a lot of different ways in which AI could be incorporated into a business, which might not be very easily measurable or easily attributable to p and l, but it's still highly, highly valuable. And so when you kind of combine well, you're expected to to be, at this stage, trying a lot of things and potentially some of them failing. But then also, you're probably trying a lot of things that maybe aren't touching your frontline operations yet because that's a very risky place to start, and maybe we're working and trying on delivering ROI in in in softer ways so so we can try. So that was, like, another thing that I as I read the study, called out to me is p and l is is definitely one thing. And at the end of the day, we're all driving, we we all want to do ROI, AI to drive that as a measurable. But in the early days, there are a lot of other ways that you could be thinking about driving ROI. I would say, like, you know, you talked about the analogy between where we are and where we work with machine learning. You can also think about it if you if if you want to in terms of, like, how a child learns. Like, as organizations, we're really learning on how this AI stuff works. The models are still getting better and better, and we still are not at the final stage of kind of, like, AI. And, potentially, when people, read the headlines, maybe they're expecting for us to be able to do the equivalent of, like, write novels. But, actually, we're probably at the stage where we're teaching our kids to, like, write their first couple words or sentences. And, that's like an expectation management thing you have to do, but you also have to your kid has to learn how to write their first sentences in order to one day write a novel. So what you can't do as an organization is say, okay. It seems like everything is failing. I'm not gonna do anything for the next ten years until everything succeeds because you haven't built the muscle memory and capabilities internally of writing learning the alphabet and writing your first words and sentences and all those things in order to one day be able to write that novel that really transports your organization and has very, very, like, exponential impacts to to PNL. Exactly. Right. It's not because we're we're not there already that we're not going to get there, and we don't need to start taking those those first steps. So the the study, of course, is much more than just that headline. And, once again, as we were saying at the outset, I think that that's where a lot of the most interesting, insight from this study is. So why don't we start digging in a little bit further? So one of the the the first quotes and graphs that we wanted to look at was this one, which really focused on the difference between the success rates of generic solutions, versus custom solutions. And so, Miriam, I I I wanna ask you, you know, how do you think about this distinction between, between the generic and the custom? And this is a theme that's gonna keep coming back here because, of course, in the study, it kept coming back throughout the study as well. But, you know, how are you thinking about, about this graph and about, you know, this this initial observation? Yeah. You're completely right. So the most interesting part of the study was not the headline number, but what those 5% are doing right and, you know, how can how can we do more of that. And so this graph, and it's I'm quite a visual person, so I asked her to put it in. There are two colors. There's a dark blue, which is general purpose LLMs. You can kind of think of this like your chat GPT or your Copilot type, models or applications. And then in the lighter blue, you have embedded or, or task specific, Gen AI as well. And what you can see is there's, like, a really, really big difference between, these general purpose LLMs and these embedded and task specific LLMs as well. Most people are trying the general purpose LMS, and that's something that I would recommend. They tend to be kind of where you can learn the most because these models are very good at kind of generic language and and learning more broadly. And so working with these generic LMs is a great place to start, and that's what was mainly investigated. And it actually turns out that these general purpose LMs were also more often, implemented as well. And I think there's a lot of reasons for that. Firstly, they're easier to work with and develop. And, secondly, they're also easy to actually just, like, get out widely into the ecosystem, whereas task specific LLMs do involve, research and development to actually build. And so I think we see more implementation of these general perhaps LLMs, but that doesn't necessarily correlate super well to where we see success, from from these applications in these and and these models. Something that kept kept coming up in the study was that, a big driver of whether or not an application would work or not was whether it had, like, memory or, like, was embedded into a task or have, like, deep understanding of the task. And that they found that there was much higher ROI when that was actually linked to, like, the business process and having memory and all of these things. And so that's, like, a very interesting thing in terms of the general purpose that allows them to be easier to work with, 80% of, people then have investigated them. But, actually, they might be lower linked to ROI. So that's this trade off of ease of ease of getting started and working with versus what actually might be a a driver of of ROI as well. Yeah. And I I mean, I I recall as well, a lot of the early conversations around Jenny and I really focused on fine tuning. And there was there's almost this, like, you know, in the sense that we need to be able to fine tune a model in order to, to get, you know, any sort of use out of it. And I'm glad that that's kind of, you know, been, been dispelled. Fine tuning can be really useful, you know, setting up a model, you know, in in doing the work to to train it on, you know, your specific content. That can be really relevant in a lot of, especially very specific context. We have some customers who are doing it in, you know, industrial context where they're essentially teaching it the language of their, their process manuals and some of this, machine generated content which comes off of their, their different assets. It can be very relevant in the pharmaceutical industry. It can be very relevant in the legal industry where there is a very specific, type of content. But for most purposes, right, and even within those companies, for most use cases within those companies, you don't need that kind of task specific, model in order to get value out of it. What I think what's easily forgotten is just how how capable these general purpose models are, which is, of course, one of their great advantages. It means that you can, direct many different questions to, to a given model, and have them handled, without having to switch back and forth. That's ultimately what's going to give you some of this flexibility within, within an agent when you're building it, is the fact that a, one of these general purpose elements is going to be able to handle different situations. It's going to be able to handle, unexpected situations which are not just programmed into, some sort of automation. And so I think that, as we progress and as organizations mature, we're going to see a growing understanding and, again, further development of best practices for knowing when to use that big general model versus that, that narrower or or more specific model. I think this in particular is a really good example of how early the space is. Two years ago or even one year ago, the only real way to get the performance you needed on particular tasks was to do fine tuning and was to do some kind of, like, task specialization. But the models have actually gotten so good so so quickly that actually for a lot of use cases, apart from some of the ones you called out like pharma and manufacturing, the general perhaps LLMs, given you give them the right business context, the right data, the right prompting, and all of these things, actually perform really, really well. And so, you know, that's a that's potentially a reflection of how early we were in the industry a year or two years. When we first started, we were actually a fine tuning platform for this exact reason. But I think I don't think we should confuse, task specific Gen AI models being successfully implemented with the importance of making the AI application task specific. Because just because the model is not a task specific model, maybe you're using OpenAI or Alarm a model or DeepSeq or something like that, they can be generic. But the way that you tune them and use them in a particular use case, The study seems to to suggest to me and what I've seen from our clients is the more specific and targeted the use case, the more likely it is to actually be useful for the end users. Have you guys seen that as well? Yeah. I mean, I think that, we're all learning very quickly the importance of really tightly tying that, that use case to the business process. And I think that, you know, that's covered on, on some of these next slides. Before we go to that next slide, I'm keeping an eye on the, the chat in some of the questions. And I think it's relevant to, to address Madhur's question, which is asking, about in test specific elements, was it the ROI low because fine tuning was not as good as reg? I think we don't wanna make, like, a blanket statement, but I think the reality is, like, if you need a, you know, one of these systems, if you want, one of these systems to give you the answer that you want from your own documentation, setting up a RAG pipeline, is a really efficient way to do that, which definitely does not require, fine tuning. And so I think, indeed, you know, we saw RAG, you know, answer a lot of those initial questions of how do I make sure that the, that the model is just giving, truthful, fact based documented answers. So just wanted to to to answer that question while we're on this slide. Well, why don't we, why don't we move, to the next, quote? And so let me just read this one off. Organizations that successfully crossed the GINAI divide. Right? And so that's from, you know, going from kind of unsuccessful pilots to successful pilots to successful deployments. They approach AI procurement differently. They act like BPO clients, not SaaS customers. They demand deep customization. And so, of course, BPO, this is where you're, you know, outsourcing some of your business processes. Right? And the the goal is in that process is not to change what you're doing, but rather have, have that business process actually be faithfully implemented by an external partner in the old days. Right? A consultancy in the old days. Still today, of course. I think this is so most of it, is getting done today. But I think what the what the authors were looking to say here, I think is a really important point and gets to, to what you were just saying, Miriam, is that the expectation, or the need for AI, which is really impactful in the enterprise, needs to be really tightly coupled to the actual business process with the way that the, the business actually works. And I think this is actually one of the really interesting things about, about Agindec AI in particular, and I appreciate that they draw drew the distinction with SaaS customers. So SaaS customers, you you you buy the functionality or you license the functionality which is available to you and you get, you know, when you get that functionality. And in many ways, I think a lot of companies have been almost adapting their processes to the way their CRM works, to the way their ERP works, to the way their, IT, management system works. Right? And what's interesting about agents is they can sit essentially above those systems, connect into those systems, and allow an organization to function in a way which is more flexible or, you know, the the agent can be more easily adapted to the actual functioning of the organization, and not necessarily be constrained to the boundaries of those different, SaaS solutions. Meaning that we have an opportunity to build, you know, a new type of software through these agents, which really conforms to the way that our business works or maybe the the way that we want our business to work. And we haven't been able to achieve yet because we have been forced to adapt to, to, you know, kind of the pillars of the SaaS, software that we've been using so far. What do you think about that? I mean, how have you been thinking about, you know, especially that topic? So I I guess I have two levels where I can comment on this. So the first level is, what I've seen work in our customers, and the second thing I can comment on is, me acting as an AI company. And so me acting as an AI company, there's a role which has been increasingly common in pretty much every single AI company has this role now. OpenAI has it. Anthropic, they all have this role. And this role is has various names, but it's typically called a forward deployed engineer. And essentially, this is different to, like, an implementation engineer. It's different to, let's say, like, a sales engineer. What this is is it's a hardcore a hardcore engineer that actually normally could be in your product team that you send out with your product to your customer, and their job is to mold your product into the needs of the customer really, really closely. And pretty much every single AI startup has ended up with this kind of model. And the reason why is for a couple reasons. Firstly, from the start up company's point of view, it's the best way to learn. So for us, it's the best way to make our product as good as possible, but really closely integrates with the client. But the second reason is that in order to actually get value from AI, you need to understand the business process. The reason why a lot of companies haven't seen success with, like, rolling out Copilot licenses, for example, is they don't really understand the business. And so sending out alongside your kind of raw toolkit or, you know, as you send your engineer, you can turn that toolkit into something that really adds value for you as a business. And as an AI company, you're selling the outcome as opposed to the SaaS solution. So this quote really aligns with how we've seen the AI companies, align with how they're delivering value. From what I've seen from our customers, the most successful projects that I've seen from our customers have actually been driven by the business units but enabled by tech. And what I mean by that is the key stakeholders are people that really understand the processes, are really able to tell us, what needs to be done in what way. They can tell us, this is the kind of ROI driven, all of those things. They don't necessarily know how to execute that on that, but then they use tech as an enabling function to deliver on that, but they really have ownership over those pro projects. Those projects where it's business first and tech enabled rather than, like, tech first and kind of, business informing are far, far, far more successful because you're actually really able to customize the technology. The issue in AI is actually not a technology problem normally. The models are more than powerful enough. Typically, the problem is a translation of business process to tech. That's what the difficulty is. And the more you can bring them close together, either by AI company sending in board deployed engineers like we do or like OpenAI do, or by having the project really be business led or and I know Dataiku, you guys really champion this model of of distributing innovation within business units. Those two are the best ways to make sure what you're building is actually driving ROI rather than just being cool tech. Yeah. Absolutely. And, again, you know, I think back to, those early years in machine learning, again, you know, going on a decade ago. And, you know, at that time, you have really smart people on the companies but who are not yet fully trained data scientists. Right? They were they were statisticians. They had, you know, different titles, and maybe they, you know, they they had just gotten the new title data scientist. And what we were doing at that at that time is we would go and we would work with them, with you know, we didn't call it a a forward deployed engineer. We, we call it just a a data scientist. But we would send one of our people out there to, to work with them to to help build out some of those use cases. The philosophy of Dataiku has always been though that through our product, we're going to enable you to bring together your internal, you know, data, ML, and now AI experts because, you know, there's there's always going to be a few and hopefully there's going to be more and more. But allow them to work really, really closely with the business subject matter experts, who know exactly what, you know, needs to be done with that expert in marketing, with that expert in, you know, in factory operations. And it's through that combination of bringing together the people who know how to leverage the technology. You know, probably some external people, those forward forward deployed engineers, you know, who can come in and really advise on how to how to use it. The business experts who really know what, what needs to be done. And then in the Dataiku view, always with, you know, that, that underlying platform in which everyone can can work together such that ultimately people can start building these capabilities themselves, which I think ultimately is gonna be really important, because there is a scale problem here as well. When you think of how many, you know, how many processes within an organization could benefit from augmentation or automation by agents, working together in a reliable way, like, it's a lot. And the only way that you both hit that scaling imperative, the relevance imperative, the security imperative is by getting a lot more people involved. That, of course, does require that, like, transfer of knowledge, to bring people up to speed. And so, like you said, right, technology can, can play a really critical role in that. And, you know, that includes both the the AI models themselves, which, you know, I can only agree that their their capabilities are so far beyond what we need right now. You know, that's not the issue. But there's also then, right, the enabling technologies, you know, to make it easier to use, you know, you know, be it, you know, running them internally, you know, doing internal, reference, like you can with double word or bringing together business and, and AI experts, like you can with Dataiku. We're doing all of that together with both. You know, that's, that's ultimately where we see those challenges being being fully unlocked. Yeah. I mean, Dennis in the comments just said, like, what the report actually kind of says is it's not really a technology problem, and it's largely an implementation and strategy problem. And, you know, that's kind of the things that we think we should be solving for in in the next coming months and years is that the tech actually, if the tech stops improving right now, we have about ten years worth of innovation to unlock. The goal for all of us is to figure out what is the best way to unlock and roll out that innovation across the organization. Absolutely. Alright. Let's go let's go on to the next quote. So here, this is a a a a really interesting one, and I think I think we all kinda smiled when we, when we read this one. But, you know, it was contrasting the fact that there were these, you know, kind of, quote, unquote, disappointing headline numbers about AI deployment. But then we contrast that with the reality of the what they call the thriving shadow AI economy where, you know, essentially, people got access to AI in their personal lives, you know, because, you know, as of, December 2022, we're all downloading chat g p t. You know, you don't have the growth rates, that they did, in their usage without everybody downloading it. And, of course, immediately finding all these useful ways to use this technology and then kind of sneaking it into, into their work. And so, you know, when I read this, I I I think that this is, this creates both the opportunity and the risk. And I think that the the opportunity here is that we see the appetite that there are for employees to do their work more quickly, to be more effective in their work, to, you know, accelerate some of the mundane parts of their job. I mean, I have a whole bunch of small tasks now that I just hand over to, to one of the chat bots and just, you know, it does it for me and, you know, that is just makes my day better because I don't have to do that, that, that repetitive thing again. But at the same time, I think that it also is a risk because it helps. I I think it's it has been one of the drivers of, let's call it, overinflated expectations. Right? Where in all of our personal lives, we've discovered, how useful this technology can be. And then there's this frustration with, well, why can't I use this fully at work yet? And my thinking on that is, really that we what we're seeing is the you're kind of this, the confrontation between really potentially transformative technology, and I do believe that AI is going to transform the enterprise, but just the reality of enterprise adoption. Right? If you've ever sold into you know, sold software to an enterprise company before, they're gonna ask you a lot of questions about, about your security frameworks. There's going to be a procurement process. Right? And all of that is for very good reason. Organizations need to be risk averse. They need to make sure that their, that their operations, which they've built up over decades, sometimes centuries, are respected, that they're not breaking anything when they when they bring something new in, especially something as powerful as, as AI. And so, I think that this this gap between individuals' personal use of AI, which can be very advanced and where they can, you know, move really quickly versus the, let's say, necessarily more, more moderate pace of enterprise AI adoption, I think this is something that we all need to be conscious of, and we need to ensure that this is not something where that, the expectation of just having been able to use it in my personal life means that I shouldn't necessarily have it immediately in everything that I do at work. Now I do think that we're getting past that. Right? We we've seen a lot of deployment of copilots of, you know, kind of internal, chat bots in the, in the chat g p t mold. But, you know, ultimately, I think where it's going to have its most impact is once it starts tackling a lot of those really specific business processes, and that ultimately is going to be a different experience as well from the, from the, from the chatbot experience. So I think that there's, you know, this shadow AI economy, which is both creating a lot of excitement, setting up a potential for, for frustration, but then also going to need to adapt, its understanding of how AI is used in the in the enterprise. Because a lot of the AI that will be used won't necessarily have a chat interface and that may you know, won't necessarily be the, the main way that you're working with it. So I think it's you know, when I read this, I see something which is definitely true, but I think requires a lot of, you know, kind of nuance thinking about what it actually means for, for the adoption of AI in the enterprise. Yeah. I, I can I agree with that? I think my favorite thing about this quote is the fact they call this a surprising reality when I don't know anyone that would be surprised by this. I think we are all seeing this, that the shadow AI economy, everyone is secretly using ChatGPT even if they're not allowed to. They're just using it on their phone or their laptop instead and and actually just, you know, doing things they shouldn't. So I don't think it's a surprising thing at all. But what it does tell me on an optimistic note is is this. I think what it's telling me is that when we put AI into the hands of people that understand understand their business process the best, we get really, really good results. And I can give you an example of this. We're working with a actually, a a Dataiku customer. And one of the use cases they wanted to do was to be able to automate, like, the when they were reviewing NDAs and make sure that all of the NDAs that they received or sent out have a checklist of clauses that they needed to have. And we were talking with them about, you know, we could build this application or or or their data science team wanted to build this application. We get on the call with the subject matter expert, and he pulls up his, personal, ChatGPT, and he's like, I already made a custom GPT for this. It took me twenty minutes, and it's perfect, but it's annoying. I can't put all my contracts in it. I can only put some in it. And he had, in twenty minutes, actually saved himself and his team hours and hours and hours of work, and what he was annoyed about was that he couldn't use all of his data in it. But what it showed me was that if you actually put AI into the hands of the people that understand the business really well, you know, whether allowed or shadow, they actually make really good ROI decisions. They actually make really good decisions of how to use that AI. And so that was really hopeful. And, obviously, shadow IT economies and shadow AI is pretty dangerous, and we definitely don't want it to be something that we encourage. But as organizations, we can look at the fact that, actually, our employees know what to do with AI when we give them AI and give them the ability to build things that make their workflows and lives better. As central IT and as organizations, we just need to figure out how to enable that distributed innovation in a way that is safe and governed and controlled. And so that's the the message that I I learned from that experience is the shadow AI economy is actually just showing we have really fantastic distributed innovation. The the job for enterprises is how to make that, innovation and, and and everything safe. And so that will have actually it's definitely not a surprising reality, but it is actually potentially a really optimistic reality. Yeah. I think, you know, that's that sums it up perfectly, right, where one of the great things about AI is that we have this ultra powerful technology now, which has a super low barrier to entry. You know, again, I'm just gonna keep referring back to ML. But the you know, there was a you know, back in those days, right, there was kind of a big barrier to being able to use machine learning, you know, if you did not know how to write Python. Right? That was a that was a big barrier. But given that this is a a technology where the interface is primarily natural language, where you can just talk to it and tell it what to do, and the learning can be so intuitive, I think we're at the cusp of potentially really unlocking a massive wave of a lot of really smart people in a lot of companies being able to accomplish new things very, very quickly and very effectively in a way that we haven't seen before. So I think that that's, that's a really key insight. It's actually so exciting. The the fact that data science projects and actually a lot of tech projects have really been gatekeep gatekept by technology departments, when actually the people that are really driving businesses forward are the people in the actual business, yeah, making the business function. And tech is an enabling function for most businesses, and AI allows us to to return to that space where tech is an enabler but doesn't have to gatekeep all of the innovation. And, actually, we can have pretty rapid innovation and experimentation in the business. And I actually like the idea that we should be trying a lot of things, and some of them generate ROI and then some of them don't as long as that innovation happens quickly enough and happens in the places that make sense. So that's a very optimistic way, to to see how kinda AI innovation might grow across all. Exactly. Yeah. And I think that, that fail fast mentality, really has its role to play here. I'm just seeing in the chat, Dennis is saying that, you know, this just shows that employees need access to these tools and able to, to scale their use, and that's exactly it. Right? I think the the the imperative now, for companies is to be able to equip their people with these tools or equivalent tools in a way which is, of course, secure and cost effective. Now I say that working at Dataiku, that's kind of our main mission these days is making sure that, companies have those tools. But I think that there is, really an exciting opportunity here, just like you're saying, Miriam, to to unlock something fundamentally new that doesn't start with, you know, people having to wonder, well, how many people do I need to train on Python in order for them to, to use that? That's a really that's a really big first step. And here we are now, and if we can just get it in into people's hands in a way which IT approves of and which remains within budget, I think that we're going to sound like a, I think we're going to see something really exciting within, within the company or within companies worldwide. Yeah. Does it actually have a potentially dystopian future of AI, right, where everyone feels like their job is being automated away and it's all very horrible and kind of, and all these things, which I I don't really think is is gonna be the case. But actually exciting opportunity for AI and the exciting vision is that people can use AI as a tool to make them their work more enjoyable, to make them better at their job, to allow them to work on interesting and creative things. And, like, that's the future that I'm way more excited by. For sure. I just wanna react to one of the comments. Stefan just said, every time I hear a seminar on AI, it sounds like Internet talks from 1998. And I can only agree. Right? And I have to like, I hate that this is being recorded, because when we look back at this, you know, fifteen years from now, it's gonna be so naive what the way that we're talking about it. It would be like we're like, every company needs to be able to build websites. It's like, what do you mean by, like, websites? Like, we're talking about it, like, before there was ecommerce, before there were blogs, before there was, like, social media. Like, technically, those are all websites, but they're totally different formats. They're totally different, concepts. And we're here, like, just, like, riffing on, like, man, have you tried out CSS? Have you tried HTML? Like, it's crazy. It's awesome. What and so I think that, Stephanie, your point is, like, a 100% true. I think it's I think it warrants the enthusiasm because, like, the Internet, you know, initially, there was a little up and down in it, but kinda became a big thing, in the end. But yeah. Totally, totally cognizant that we're we're at the very beginning of this and, yeah, going to be very, very embarrassing to look back at this. I already have it from webinars I did two, three years ago, where I was talking about, like, how's, like, this thing called RAG. I'm like, do you know all RAG is? It's just, like, giving the model a bit of extra information that makes it know more about your use case. And I'm like, well, obviously, we would want to give the model kind of relevant context. But at the time, it was kinda like, oh, wow. This really boosts the performance. And I look back, and I'm like, this is embarrassing. And that was only, like, two, three years ago. Yeah. It's gonna go so fast. It's it's super exciting. But, but, yeah, we have to be cognizant to where we are right now. Alright. Let's, let's move on to the next quote real quick. And so this one just kinda brings it back to, process specific customization, that they saw, you know, more success, rather than others that were just looking at peer benchmarks. And so there's two things in here that I, that I that I that that really struck me. One is, again, that process specific customization that we, that we touched on. But I think I like this quote because it emphasizes the process aspect of this, because I think a lot of the applications here, it's, it's useful to think of them not just as an agent, but rather as a complex process that might have an, one agent over in one part of that, doing, you know, some part of the work, handing that work off to maybe another agent, but then handing that work off to a human being who is doing part of the work as well. And so I think it's, that's going to, I think, feed our understanding of the way in which these systems need to be built. It's not just one agent, but it's really a process which is going to integrate agents of different types, along with other assets, things, you know, data data sources, various tools that those agents can use, machine learning models, and so on. And also, of course, the, the humans themselves. And then there's the other, the other aspect which is, benchmarks. And there was a question earlier as well around, like, benchmarks for ROI. And it's not that we're dodging that, that question, but rather it's just like that's a hard question to answer because I think we have to think of we're going to have to think of new benchmarks for, for the way that we think about the performance of these systems as well as the value generated, by them. Because, again, they are they are quite different. And I think that, you know, the, the fact that we're building this in real time and expecting ROI and expecting, you know, you know, really solid assessments of performance. I think it's just another indication that we're still early in this process of building these capabilities in the enterprise for real. If early in the process of settling a lot of these best practices. And so I think that, you know, this is something that we are all going to need to, to keep in mind as, you know, we're getting requests from our leadership about, you know, what's the benchmark, what's the ROI. That look, it's just really early, and we need to, you know, we need to start moving this forward. Yeah. I I do also like the quote process specific customization. Some of the best success that I've seen with our pilots is when, with with with our clients and, like, their internal pilots is when they take a process and they break it up into how many steps make sense. And we say, this is low hanging fruit. This is low hanging fruit. This is always gonna be human, and and this is where we can help. And then do those projects kind of one by one and then slowly chain them together. And the reason why that really works is because you first you take off a project that's, like, manageable, and you're pretty confident that, you know, they know how to execute and that and that, you know, has been has been done before. But also, if you are optimizing a process or a part of a process, it's really, really easy to measure that ROI. As an industry, we love benchmarks, but our benchmarks are typically pretty abstract and tend to be very badly representative of real life workflows and real life things that actually matter. And so actually understanding your specific process and your specific use case and optimizing for that is the best way to generate kind of measurable ROI. I do often think that the companies who are going to see some of the most success from AI transformation are the companies that are the best at training their juniors. Like, if you're really, really good at writing down a process and understanding what goes into it and what the steps are and what data you need for x y zed, then, actually, you've got, like, a a a process ready built to to go and make better. The people that struggle are the people who are trying to use AI for more of a vibes based process, and that is is much more difficult. So people that have strict rules, we do x, I said, we get this from these kinds of databases. Even if it's not like a prescriptive task. But, you know, those those structures to process tend to do much better from my experience. Yeah. I think that's a, that's a really useful insight. And I love that you called out, like, the role for junior employees and all of this, because I we've seen other headlines recently about how hiring of juniors is decreasing because now we can just have AI do that. But I think that you've totally keyed into something very, very specific that those junior employees can really be helpful on is, is that process of translating that a a business process into Inogenetic workflow. You know, they're they're used to working with this technology, and, you know, they're they're fast learners. You know, not like old people like me who I can barely learn a new thing these days, it seems. But it's, you know, it's a, it it it it is a positive vision for, you know, for what, what what entry level work can, can be in the future because we're going to need entry level work. Right? We're not the last generation to ever need a job. I don't believe so. Right? I I think it's the one of the biggest fumbles I've ever seen a collective society make is this decrease in graduate hiring and employment. I feel it's it's so so stupid. The the best predictor of how good your workforce will be in five or ten years is how good the people you can hire now are and how good the entry level people you bring in now are. Given how difficult the entry level job market is, it's actually a really great opportunity for companies that want to level up their kinda average quality of their workforce to get the grads they never would have been able to afford or never been able to get in in previous, generations. Bring in the best grads now because they're available, and then use that to create your kind of your standard for five, ten years time. So I think it's a huge mistake that the society is making just from a business perspective, let alone, societal. Exactly. Right. No. It's such a positive vision and so important. Alright. Last, last quote, and it it's a doozy. So this one was so good that we, we just, screenshotted this one. But, basically, you know, people will read it, but what we're what we're seeing here is, the importance of decisions that are being made right now. You see in here, it's talking about this eighteen month horizon, of when decisions key decisions are being made about which, AI technologies organizations are going to be working with because and it correctly identifies it down at the quote in the, in the bottom. Once you start building those specific processes in a particular way, there there's going to be high switching costs. I won't call I I don't think we should call it lock in because, you know, you never really locked into anything. But if the the switching costs are too high, then indeed you're going to you're going to need to be stuck with that. And I think that that's yeah. I think there is some truth to that. I think that there's, I know that organizations that Dataiku is, speaking with, there's a lot of questioning about, you know, are we you know, should I be using models from this provider or that provider? Should I be building in this framework or that framework? Dataiku's approach to that is always to equip you with a platform that says, well, use just whatever you want and, you know, you you'll be able to maintain that flexibility into the future. But I think that there really is some of that, some reality to to to these decisions here that the the choices that you that you make now, well, yeah, there is there's going to be some path dependency. So making a small decision right now about the way that you build, that might lead to, to some, yeah, higher switching costs and the need to make, commitments, that, that ought to be thought about right now. Mary, what was your reaction when you, when you saw this and thinking about that, that, you know, the importance of decisions that are being made now? So for this, I the reason I picked out this quote is I I think this is something that's really underappreciated of. If you're building things so closely tied to your business processes, it's very difficult to unwind that later on. And for me, this comes down to what are the factors that are involved in a build versus buy decision. And, actually, I think I've heard Kurt talk about this in the past. So what are the kind of AI processes that we're happy to buy off the shelf that maybe we're happy maybe for our HR platform. It's not like, you know, it's it's been this critical, but we don't build our IP around that HR platform. So maybe you're happy to buy and be locked in for a period of time now. But the things that maybe you don't want to be locked into an external vendor for, at least for the actual business process and application, is probably those parts of your business that are really unique to you and that are your real value drivers. So, for example, if I'm a pharmaceutical company, I probably want to be building my key IP around AI and how I'm using that in drug discovery myself and internally, and so I'm not beholden on someone else to do my r and d for me. And so for me, I think it's a build versus buy decision. I think I mean, cut you kind of use the was it commodity? What's the language you used of? Yeah. We I like to draw the distinction between, you know, kind of commodity, processes and differentiated processes. You know, there's there's some processes in your business, where you need to be good at it, but it's not how you differentiate yourself from the from the competition. Right? This is a lot of back office processes. You need that to be efficient. You need to onboard new employees efficiently. Right? You need to, you need to run your finances, efficiently. And a lot of these you know, a lot of software providers, a lot of those SaaS, providers that are specialized in those domains are going to, are going to provide agents that, you know, assist with a lot of those very common processes. And that's that's probably a very good, buy option for exactly the reasons you said, Miriam. These are kinda commoditized. Everybody needs to do them, so you just kinda level up with everybody else, and now you do them all more efficiently. That's great. But that's in contrast to those use cases, those processes where you are seeking to be better than your competition, where that's actually the IP that's going to make you more successful than them. And so, you know, you mentioned drug discovery for pharmaceutical companies, you know, maybe it's a supply chain optimization for, for manufacturers. Maybe it's customer onboarding for a financial institution where you can do that more effectively, more quickly, and deliver a a better customer experience. I think that there's a, there's a important distinction there, between, what is really critical to the way that your business operate operates, where you seek to differentiate, and where you hope to gain competitive advantage. Yeah. So So we're kinda coming up on time here. I'm the next slide is simply asking for questions. Now I do, I do think we've been doing a pretty good job keeping up with questions as we've, as we've gone. I'm just gonna click over and see, if there's some new ones that have popped up. Oh, Oscar, Oscar Miguel has a really interesting one, here. What if you're working in creative processes, therefore less deterministic and more as a playbook or toolbox? I think that that's a really interesting question because a lot of you know, there is the question of, you know, what what degree of, of creativity do you want to allow an AI system to have? So there's certain processes in a, in a business where you actually don't want a lot of creativity in it. You want it to to follow a, you know, a set of rules, and you want it to have maybe some flexibility, but you don't want it to get creative. Now that's in contrast to, like, a truly creative profession where, for me, I think that, you know, at best, AI is going to be, an assistant, you know, to help you, achieve your creative vision more quickly, but it's not going to be the source of creativity. Personally, right, in this, you know, it's more of a philosophical, discussion than anything else, which, you know, Marion, you know, we're we're happy to get into. I do think that there is something uniquely human in that, in that true creative creativity process. So I I I yeah. I my belief is that for creative processes, should be looking for, you know, kind of acceleration, but not outsourcing that creativity to, to the AI. What do you think, Maryam? Yeah. I it's you're never really going to or at least less clear how you completely automate a creative process. I kind of think of these kinds of things. You're really thinking about it as, like, a copilot and assistant as something that can maybe help you iterate and and learn faster, but it's definitely definitely less easy to measure ROI in in that kind of creative process. So that one is a bit more of a vibes based situation, but also the person using the tool should be able to judge for themselves whether they're getting value out of using them. Yeah. Yeah. Really good. It's a really good point. So there's another question as well in here about, also from Oscar Miguel, asking about who owns the training. Is that intelligence known by the models, that can be reused elsewhere, or should those learnings be owned by the company to reduce the lock in? So one thing is that, depending on your subscription and how, yeah, how you engage with the the model providers, they they don't use your, your interactions with them necessarily for, for retraining those models. So that's one of the, you know, the main advantages of a lot of the enterprise subscriptions with them. But I do think that there is really high importance on organizations being able to set up systems where they can internally capture feedback on what is working in one of these systems, and then quickly feed that back in and address the root cause of that, whatever that feedback is. Right? You know, assuming that it's a problem. Because I think that that's where organizations end up with a really positive dynamic of improving these systems quickly, such that they can, they can give, better results. Yeah. Actually, one of the key things that the report pulled out time and time again is this idea of, like, memory and specialization and, like, constant improvement of the application. So it's absolutely critical that as an organization, when you're delivering your AI apps, you're building in pretty robust logging, monitoring, feedback collection, so you are actually able to create that, feedback loop as well. Yeah. And then just one last question, before we wrap things. Elizabeth asked, you know, what are the most critical roles in an organization, in order to adopt AI safely? I can't emphasize enough the importance of, AI governance, but also shifting the understanding of governance from just, you know, kind of check boxes and compliance, but rather creating a process by which you can ensure that the AI that you're developing from the ideation stage is going to be aligned with, your, internal policies, with any regulation that you might be, subject to, as well as, of course, just your your values about how you believe that AI should be used in your, in your process. So for me, there really needs to be somebody responsible for AI governance in the organization. And without that, not only are you running risk, but you're probably not actually going as fast as you could be either. You know, guardrails, there's a reason that there's guardrails on a racetrack. It allows those drivers to drive more quickly, because they know that the, the consequences of a mistake won't be so terrible. I think for me, often when we hire technical talent, this is sometimes and I'm saying this as someone who works with a lot of technical talent and, you know, being an expert as myself. We often hire people that don't really love speaking to other people. We like people that, like, sitting in dark rooms and coding. And, actually, now is a really, really good time to build up an engineering org and encourage your engineers, to get more interested in the actual underlying business process that they're working on and work really, really closely with business units. So, forward deployed engineers is really working for companies like AI companies as they look to integrate their technology into businesses. I think internally, for deployed engineers as a role makes a lot of sense, within companies themselves as well. Absolutely. I can only agree. Well, thank you, Miriam. This has been fascinating. To everyone online, thank you for all your engagement and all of your questions. If you wanna learn more about Dataiku or Double World Word, you've got, well, either Google or QR codes, right here for you. But we really appreciate the the the conversation today, and we're looking forward to to having you all on some of our future sessions as well. Thank you so much, Scott. Always a lot of fun.