Video: The AI Accountability Reckoning: 600 Enterprise CIOs on Proof, Pressure, & Performance | Duration: 3644s | Summary: The AI Accountability Reckoning: 600 Enterprise CIOs on Proof, Pressure, & Performance | Chapters: Welcome and Introductions (6.16s), Enterprise AI Landscape (201.67s), AI's Career Impact (392.795s), AI Adoption Challenges (781.335s), Explainability in AI (890.675s), Agent Accountability Frameworks (947.66s), Agent Accountability Challenges (1051.65s), AI Vendor Flexibility (1351.025s), Building Flexible Stacks (1670.795s), Interoperability and Standards (1771.615s), Cloud Interoperability Challenges (1878.22s), On-Premise AI Investments (1976.195s), Multi-LLM Strategy (2090s), LLM Evolution Challenges (2198.31s), Governing AI Sprawl (2303.91s), AI Data Concerns (2499.95s), AI Production Challenges (2631.64s), AI ROI Challenges (2804.03s), ROI and Alignment (2986.725s), Balanced Portfolio Strategy (3204.855s), Governance and Decision-Making (3327.375s)
Transcript for "The AI Accountability Reckoning: 600 Enterprise CIOs on Proof, Pressure, & Performance":
Alright. Good morning, everybody, or good afternoon, depending on where you are in the world. Welcome to our conversation here this morning. Excited to get going. Excited for the conversation here today. Let's go ahead and introduce ourselves here real quick. My name is Connor Jensen. I lead the field CDO team here at DataIQ. Been here for more than seven years, worked with customers around the globe and all sorts of stuff, and actually came from the customer side. So I've sat on the other side of the fence, now a little while ago. But it's exciting excited to be here to present with Treveni today. Treveni, if you wanna introduce yourself? Hi, Connor. Hi, everyone. My name is Treveni Gandhi, and I am the senior manager for Responsible AI at Dataiku. I have been here longer than Connor by. only a few months, maybe six months. So, yes, we have we are Dataiku veterans. Very excited to talk to you today about, what it means to be accountable for AI. Yeah. Super excited for the conversation. And it's. Karini and I used to work on the same team here many years ago. Don't get to spend. as much time together talking about these topics, these days, so it's exciting to actually get to hang out with, smart friends and talk about cool topics here. So, Yes. let's go ahead and dive in. I'll actually sort of go back to this one and give a little bit of framing of what we're talking about here with this AI accountability idea. So what we're going through here today are some of the key decision points that came out of a recent survey that we did. We interviewed 600 different CIOs on all sorts of what's they're experiencing for the AI. I, of course, forgot to say this as we're starting. This is all being recorded. Of course, we will send that and the slides out afterwards. I've already seen a couple of people introduce themselves in the chat. Please feel free to throw a introduction and a hi in the chat, and we have a q and a over there as well. So if you have a question, go ahead and throw that, over there. We'll keep an eye on those while we're going, and we will definitely try to take as much time as we can at the end to get through those questions. So sorry. I'm a bad housekeeper. It's alright. That's why my office you can't see my office. It's a good thing. So, anyways, so going through the the survey here, what we really wanna talk about is some of the key points that we saw coming out of that. Again, it was 600 different CIOs from all different regions around the globe, primarily larger scale enterprise, institutions, both sort of like the big huge fortune companies, but also, you know, smaller and medium sized enterprises. So, hopefully, the results here, the things that you're hearing will reflect your own experiences in your organizations and, give you something to go on as you're going into this next year. If you're not familiar, just real quick, I'll do a quick introduction of Dataiku, then we'll dive in. Dataiku is really focused on what we're seeing as this growing divide between the enterprises who are investing and the enterprises who are actually being successful, with their AI investments. Every enterprise has data platforms, old, new, cloud, on prem, hybrid, all over the place. We have enterprise apps. Some are SaaS. Some are on prem. Some are hosted. There's all these different sort of formats out. And then especially over the last couple years, but really over the last decade or so, a huge, surfeit of AI services and tools that are coming from some of those platforms and applications and also dedicated AI tools and platforms, especially in the LLM and generative AI space. And on the other side of this divide, there are some companies that, yes, they have those things, but they also are taking their institutional expertise, scaling it. They are creating orchestration across all those different platforms, and they are doing so in a way that it has real good, understanding, visibility, governance around what is happening there. And these are the pieces that we really see as those key success factors is to really be successful with your AI investments. You have to nail all three of these things. You have to have everybody in your, everybody in your organization using AI, working with AI, helping to develop AI because you have the expertise in your business, and scaling that out to impact how are you using AI and how then using AI to scale that expertise is the key is the key piece of how you're working with people. It's not people or AI. It's not automation. Right? It's how are we doing these things together. It's also orchestration. These any enterprise has an extremely complex landscape of tools, technology, processes, everything, and we have to be able to orchestrate across all of those so that data sitting in one application can use AI sitting from a different place and I'll do so in a way with the third piece, governance. Right? How do we know that we have no blind spots? We don't have people out there doing things on their desktops, in Excel, data scientists or coders building things that, you know, doesn't see the light of day, especially the concepts of shadow IT, shadow AI. How do we make sure that we know who is doing what with all these tools? And it is nailing all three of these together that leads to AI success. We, of course, at Dataiku, this is a thing we have learned alongside our customers for the last thirteen years, and we build a tool and host a platform that is really built for everybody. It has different ways of working with it, both your sort of business users and some technical users. It is an orchestration engine that works across your existing landscape, with governance built in, from the very get go. So we hope to, you know, be able to sort of support across those three things. So if you didn't already know more, know us as a company, hopefully, gives you a little bit of the perspective of where we're coming at in this, in this world. So without any further ado, let's go ahead and dive into these key things that we heard from the CIOs, in this survey. As I said, we're gonna send out these slides afterwards, so we're not gonna try to go through all of these at once. We're gonna just dive straight in, and go through them one at a time. So I will hand over to Tarini to start taking us through these. Yeah. Thanks, Connor. So one of the first sort of decision making questions that CIOs, have been asking themselves is really what are the consequences of AI going wrong or backfiring or not delivering that value? And 90% say that their professional reputation or career trajectory is going to be shaped by their success with AI. And it's it's interesting to see just how important this career and consequence step is for CIOs. It's almost like a brand new, kind of metric for them to be to be evaluated on. Connor, I'm curious. I know that you've had a lot of experience in the past from, like, AIML, leading AIML teams and everything. Do you feel like the the pressure was as intense then as it is now, or do you think the pressure has become a little bit different? No. I think the pressure is definitely different. You know, I was really fortunate now just about fifteen or so years ago to be working at a company where our CEO was really invested in data science and machine learning and sort of like you know? And so I was that was sort of how I ended up in this world was, you know, switching over, being asked to help lead and build our data science initiatives. But we had a CEO that was very unique in that he was really, I'd say, way ahead of his time, in my experience. And in some cases, sometimes sometimes I'm talking to people today who are, you know, just now really have that same sort of mindset. And but I he was the one putting the pressure on, to be super honest. Like, it wasn't the market, and it wasn't the board putting the pressure on him to do that. You know, he really believed in that and and sort of helped to push us as a company to move relatively early, you know, in in this space. But that was unique in my experience. Right? And as we've had these conversations, I feel, you know, there's a certain aspect in this world of data and, you know, data and analytics, machine learning, however, you know, we call it lots of different things over the years that feels like the same sort of conversation with a new name every two, three, four, five years. Right? Yeah. And it's you know, it comes and goes with popularity, and it's you know, big data is a big thing for a little bit. And then everybody's like, okay. Whatever. And then, you know, then it's machine learning. And it could so it feels like the same conversations until this last two, three years. Right? And I think, you know, our, some of the people who have been at this a little bit longer than I have talked about this in terms of the change similar to what they experienced when the Internet first hit or when cloud. really sort of took over. Data. has always been a thing we've talked about, but, like, it hasn't had this same level of, if I get this wrong, that will be the end of my my job or. my career. Yeah. And I think that it's from some of those people who lived through the Internet and cloud changes, Yeah. or more to the point sort of came came into CIO sort of territory during that time period where I think before people were really dismissive of, you know, the impact that could come from these technological changes, AI feels like one where people really got on board. Sometimes it almost feels like too quickly, but it's definitely like the I mean, the the it's hard to argue that it will have this much impact on a CIO's career, and it has never felt like this to me before. Yeah. And, you know, it's interesting that you say that in terms of it it's never felt like this for you before because the report shows, like, ninety five percent of CIOs are briefing their boards on AI performance quarterly. Yeah. Right? I don't think that ten years ago, you were briefing your board on data science outputs, quarterly maybe or, you know, whatever it is. And so for sure that I think that don't even think. we even had anything interesting to say quarterly. It's. super Like Well, but that's this. this is moving so fast, and. the ability for companies to you know, we'll get even more into that. But, like like, Yeah. I I have you know, I led monthly steering committee meetings with a whole bunch of the c suite at that organization. And, know, we were working really hard and doing a lot of stuff. But in terms of, like, something that was material, I I I mean, I did present to our board, like, twice in maybe three years. Right? Like, you. know, they were interested. I don't wanna sort of label it as curiosity. I think that underscores it. Like, they had spent they'd given us a lot of. money, so they were, you know, obviously sort of in, but it was not a thing that was, you know, their quarterly board meetings where, you know, my boss or I were going to present. No. It was annually, Yeah. you know, at best. Yeah. Yeah. No. It's it's it's really interesting to see how this is changing the the script for for everybody, not just people who are using AI in their day to day, but also the CIOs and folks who are in charge now kind of at top. Yep. Great. Yay. So let's let's keep moving. I know that the the second one the second sort of career making decision is really all all my jam stuff I'm really excited about, which is that the explainability decision. Right? Can I explain and defend AI outcomes to regulators, customers, and the board? Right? And so we see that CIOs are already finding a gap between their what they've deployed or what they're what they want to get out there, and the, you know, the actual desire of what they wanna get out there. So not being able to explain what they're planning to put out into production is actually delaying for at least 85% of these CIOs their productionalization of AI. And really interesting to me is that more than 50% think that insufficient AI explainability could trigger a crisis, right, that that erodes customer brand customer trust or brand credibility. Alright. And it's and it's funny because customer trust and credibility is not actually all that new. Right? But now the shape of it is different, and people are becoming more attuned to, hey. This is AI. There's AI in everything we're using. I wanna know when AI is being used. I wanna know what that looks like. How are you making this decision about me? And and it's actually becoming more and more relevant for for leadership. I don't know, Connor, if have anything you wanted to add on that. The the thing that I'd say that is has it feels a little different in this to me where, like, a, one, this has been something we've sort of worked through in the the data science world for quite a while of, like, can I explain. what the algorithm is doing, etcetera? That's harder to do in the Gen AI world, and I think everybody knows that. And, you know, sort of the nondeterministic nature. of the models is tricky. But But the other thing I think that's really different, especially from a CIO perspective now, is, you know, I I sometimes like to make the joke that nobody came in on Monday morning to their boss and said, I was playing with this dashboard over the weekend, and it's so cool. When are we gonna roll out more dashboards here? Right? Like or, Mhmm. you know, data science models, machine nobody that was just not that was not there was no commercial aspect to what we were doing from an enterprise data perspective versus how AI is now. And so that is completely different. Right? People are coming in with, you know, AI tools on their phone. They're coming in after playing with it in their sort of own personal life and saying, how why you know what? Why can't I get access to this at work? And so the CIO is no longer the arbiter of what those tools. are that's saying and and, honestly, sometimes, we felt like the pushers. We're trying to get people to use this stuff, and we weren't pushing anything until we had a good idea what the explainability and those things were as best we could. Right? It was always a little bit of a gap. But now everybody is trying to do it, and it's coming so fast that that's. scary. Right? So I. like, that like you said, this is not a new topic by any means. Right. But the volume. and the pace of change here, I think, is what makes CIOs nervous, and and it's good. Yeah. Good. Like. Because it it's rapid. Right? Like we said it just prior. It is so rapid. Like you're saying, everybody has access to tools on their phones, on their computers. And so if I, as a customer, am now interacting with something that I know for a fact is AI, I want to be able to understand why this company is choosing to use AI or choosing some sort of decision about me because of that. And so while maybe explainability wasn't as interesting or relevant ten years ago when you were getting served personalized ads, maybe now it's much more because everyone has become a little bit that a little bit more attuned to it, Yeah. or a lot more attuned to. it. Yeah. Yeah. It's a explainability is one of, I think, one of the most critical aspects. Like, if you can build an AI system that explain that is explainable, you are far ahead of your competitors for sure. But it's hard to do. but it is hard to do unless you have Dataiku. So moving right along, you know, the the third decision that we that we talked about with CIOs is this agent accountability decision. How much of the accountability and, sort of business critical workflows do we want to turn over to agents? And if I do decide that I'm gonna give over these these workflows to agents, can I monitor them? Can I prove that there's the right human in the loop? There's the right escalation accountability when when it when it becomes mandatory. Right? And so, we see that actually almost all CIOs are saying that AI agents are actually embedded in their enterprises in some way, which is really fast, right, like we like we've said. But because, because of that, they need to have formal accountability frameworks. Right? So over two thirds of our CIOs, really expect to see those accountability frameworks and audit reports. And some of some expect that this is gonna actually be coming from industry or government regulators within two two years or less. So while everyone's very excited about getting agents out the door and putting them into business critical workflows, there will be a sort of accountability reckoning, coming, if not from government, at least industries. And I think we're starting to see, whiffs of this in financial services, in a little bit in the health care space. And I think I think those industries that have traditionally been highly regulated are going to continue that, sort of expectation, which is why I think our CIOs are also kind of thinking that that direction. Yeah. I think, you know, this is this is a a, like, fascinating topic to me. Think first, it builds off the previous pieces. Like, how do you. keep how do you create accountability of something you can't explain? I mean, like, you just can't. Right? So, like, that's we can't do this if we can't do that. So that's, I think, already, like, a hard point is it's, like, two sort of failure points in a row. But the thing that I find really, like, messy and and, like, really and and will or more sorry. Will be really messy is this idea of sort of the agent accountability of, like what what does that exactly mean? And when we. say, like, you know, if the agent made this decision and that decision violated some sort of rule. Right? Well okay. Was there a human in the loop? And somebody took a recommendation from an agent and, you know, didn't think it through and sort of pushed it in. Okay. Maybe it's a little bit clearer. Is it an automated thing, right, where we're talking about these, like, operational backbones and workflows that are just happening? Okay. So I bought that software. This is an agent that we've purchased. It's, you know, come comes off the shelf. Okay. Most software indemnity steps means that that company is not liable for, you know, your use of it. So you're on the hook for technology that you don't own or build or maintain. Or maybe it is an agent that somebody at your company built, but somebody else is using. Is the user accountable? Is the person who built the agent accountable? Like, there are so many layers in, you know, sort of each of these. If the agent got bad information because somebody put, you know, the wrong data into the program, into, you know, like, a field or something, is it the person who entered the data? Like, none of these are questions that we've even started to ask or hey. Or not. go, ahead. Go. ahead. I was I was gonna say that, yes. For sure. Like, these are new layers of questions that we need to ask about our our agents. But I think what I would I would say to these CIOs, right, is not just, oh, yes. Sit around and wait for regulators or governments to come up with frameworks for you. You already have the frameworks. You already know what to do. We already deploy software and technology out into the wild with robust guardrails. And agents, LLMs are not any different. I guess, not any different in that they also require frameworks. They also require guardrails. Maybe they require different types of questions that we ask upfront or different ways that we build. But the the notion of being accountable to if something breaks is not new to any enterprise. Right? Like, my my husband is a software engineer. He gets put on call every four weeks, and he'll get, like, the midnight buzz or whatever that something's broken. That's no different here. Right? But now it's a question of when do you alert for a break, or what are the very strict controls you're putting into place around that system? And did you think about that before you even started building? Well I think think. the thing that becomes really tricky in this sort of especially from an accountability perspective, and this is this is, I think, the big missing piece in the industry right now from a sort of agent AI perspective is traditional software works or it doesn't work. It's kinda binary. Right? Like, know, I pushed this code, and it broke something, and, you know, now we have to fix it, or it went down, whatever. But they're deterministic systems. Right? So if this, then that, and it worked or it didn't at the end. I would the challenge we have right now is not figuring out well, it's also figuring out is an agent functioning, but is it functioning in the way it's intended to? Because we don't have that same a equals b equals c sort of logic built into these. And so far worse than an agent that stopped working over the weekend and somebody was on call and had to come in and figure out what was going on is the agent that kept working but went rogue and. starts making. the wrong decisions, especially either things that are impacting our, you know, our customers, our reputation, or bringing us afoul of loss. Right? And so that's. the nuance here that I think is exciting, but but it's gonna. be a challenge and is, you know, to sort of back to that, you know, first slide of career and consequences. You know, if you're the CIO, you are probably operating under the assumption that you are ultimately gonna be held accountable for these things that. are being pushed up. They are a tech sort of, like, It's a. it's a tech thing. Right? Yeah. And that and I think that's where, again, going back to your existing frameworks is important because that's where you know that you've already started having conversations around what does compliance mean from a legal perspective, what does good security practice mean from, like, IT cybersec? And you don't need to reinvent the wheel, but you certainly have to extend it, right, or amp it up to to account for new new issues within agents. Absolutely. Yeah. Yeah. Absolutely. Okay. Great. So moving right along because we've got so many decisions to talk about. I think the the next one is interesting. Right? Because this is about that broader tech debt. So do we so the stack flexibility decision. Do I keep my AI stack modular, or do I accept lock in, regret, and reversals? Right? Because, you know, there's two major players out in the LLM space or the frontier model space, and then there's, you know, x number of other people. And I think that, like, there there is a very, a strong sentiment that they don't want to have, you know, a vendor lock in. And 74% of CIOs regret one of their major AI vendor platform selections, which is which is quite a bit. And? 40% think that vendor lock ins or LLM pricing changes are going to impact their AI budget. Right? Because if you're relying on one or two main models and that model's price goes up, well, now what? And I think this was an interesting question. Right? What would happen where the AI bubble burst? Right? This is a constant point of conversation on LinkedIn and beyond. And I. think it's it's interesting that a lot of folks would say that their company is gonna experience disruption, their own jobs will be at risk, and even the company's survival might be at risk. So it's it's very interesting in that we've put a lot of eggs in this basket, but, like, we also wanna make sure that that basket is well protected and that is not subject to you know, if part of it breaks, then we're all we're all out or our eggs all break. And we wanna make sure that it's a well sort of protected and not sort of critical kind of situation. Yeah. I, Yeah. you know, I I think part of what's interesting about this that, again, feels different than sort of historical changes is that there's, so many viable solutions or so many apparently viable solutions on the market so quickly. Right? You know, if. you think about sort of, like, from, yeah, you know, the sort of Internet phase and how that came out of the cloud phase. Right? Is that you sort of have, like, one or maybe two big dominant companies very early on, and then things started to, you know, fragmented the the markets involved and matured. But, you know, here, both between, obviously, the large Gen AI companies and the fact that, you know, obviously, just OpenAI and Anthropic are huge and growing, but, you know, that's primarily from an American perspective. There's other companies in other parts of the world. But that also every tech company you are already working with now has AI products that they would like you to use as well. That's really unusual. And and and I feel I feel bad sometimes when I'm trying to, like, figure this out. Because I remember my own experience of, you know, being new. to this sort of tech tech world fifteen years ago at the early days of, like, big data and and sort of data science and things like that and trying to absorb all these different players in the market and all these different nuances and and that. And, you know, I think we were counting products in the, like, maybe dozens at that point in time that I was trying. to learn. And today, I weekly am on a. call with, you know, a customer or, you know, a potential new customer or, you know, a CIO that says, well, how do you compare to x? And I'll be like, I don't know because I only heard about x five seconds ago. So. I have no. idea. Right? And it's like, sometimes they're like, oh, actually really a neat thing. And I could see how that you know, that's a feature of ours or maybe it's something. that's completely different or sometimes it's just paperware. Right? And, like, trying. to absorb all of that where, again, it's not just what the CA was going on and researching, but you've got your business folks. You've got desk users, people coming in daily, weekly. Hey. What about. this? What about that? What about that? And so. we know that this is moving so fast that you can't be locked into some of these early stage things. Right? Like, every. CIO experienced that before. Yeah. And so, you know, how do you but you have to make bets. Like, you can't you. can't just let You have to make. bets to make decisions. And so it is this really, like, tricky balance of I have to move. I have to pick. a vendor. I have. to pick a couple vendors and move forward. How do I make that in the the how do I make the best decision I can right now in a way that eighteen months from now, I'm not really it. vision. Yeah. And a. lot I mean, every you will I have I I made tech purchases that I regretted within eighteen months. Right? So I don't think that this is a new problem, Yeah. but it's a it's sort of a it's a growing problem just because of how. much overlap and difference there is in this space. Yeah. Well, So think that's where the hedging the hedging of the bet comes in. Right? Where it's like, okay. I've got my stack flex I need to be flexible. Right? I'm not gonna only depend on these two frontier models, or I'm not only gonna have, you know, a single vendor for all of my my third party LLMs. I wanna be able to swap in and out. And not only just from, like, a covering the sort of potential fallout perspective, but also from a general, like, usage. Right? Maybe. some models are better for certain tasks than other ones. And I want my my builders and my developers to have the ability to choose. And so, like, having something like the LLM mesh, you know, through Dataiku where you can just swap in and out which which models you wanna permission or on prem models you wanna build or whatever it is, that actually starts building in that flexibility for you from the beginning versus saying, we're picking one thing, and we're going that direction all the way till it breaks. Yeah. I think? so. the way that I've always, I mean, I shouldn't always. The way that I have talked about this over the last few years and sort of my understanding of the space has has really matured is my framing for what the right stack is is typically the fewest number of tools that serve the most people. Right? Mhmm. There is no perfect stack. And even if there, in theory, was a perfect stack today, in a week, it'll not be the perfect stack anymore. And every enterprise also has its own different place. And so for me, it really is sort of building, and there's the question that sort of came in through the chat. So, you know, the, like, the thought process for me in terms of building a stack here is very much what are the sort of, like, major tools that I have access to, whether it's from, like, the hyperscalers in the clouds or things like Snowflake or Databricks that serve up, you know, thirty, forty, 60% of my needs. And then sort of build from there out into those additional components where you're trying to keep it as minimal as possible. You 100% don't want all of your eggs in one basket. But it is really sort of, okay. What is the tool that is gonna do 50% of the requirements that I'm looking at from an AI perspective? Great. Right. Now what are the gaps? What's the tool that picks, you know, the next, you know, 20% for me or 30% for me or something like that? That I think, you know, is is a framing for me. And then the other thing that I really look for, and you mentioned the LLM mesh or whatever, is the interoperability, right, is how much proprietary, you know, like, closed end solution tech am I bringing in? Again, you can't it's almost impossible not to. Right? There is there are certain aspects of the things that you're bringing in, you know, LLMs or, you know, a public API a public AI vendor. Like, you don't own those things. You know? They're sort of, like, you know, things that you just sort of have to accept that's coming with those. But are they interoperable as much as possible? If this one stops working for me or the company goes bust or whatever, can I swap something in pretty easily, or is that company very much sort of, like, building a walled garden, and they're making it so that it's very hard to come out? I think one of the things that is really so there's sort of, like, two things, and it's it's a philosophical question for me as we sort of go through this is, you know, do we end up in the world of the Internet where there are very clear stand protocols? Everybody has HTTP. It's stuff like JDPC drivers and the data world and stuff where there are, like, clear standard protocols that the vast majority of companies just end up sort of adopting. Maybe that's MCP. MCP is evolving into that. I think there's still enterprise. questions that we need to sort of solve there. A. two a is interesting, and I talk about it a lot, but nobody is sort of really using it yet. But, you know, do we come into a place where it's some sort of Internet like protocols, or do we find ourselves, I'd say, a little closer to the world of the clouds where people are building sort of, like, big nation states of as long. as you're in the system, everything works well together. Well, or, you know, is it is it that? And so, you know, from a if I'm a CIO, if I'm a buyer, I'm pushing for that interoperability because I need to be able to have some flexibility built in. I can't go all in on one vendor that will be my all. That doesn't work. We know that. Right? Like so I. think that there are so many questions that you go through this. It's a really trick it's a really tricky one, though. Yeah. No. And I think I think that flexible I think it's it's it is that it's a continued hedging. Alright? It is just like it is unfortunately gonna be a continued hedging because I don't think we're gonna go into an, Internet protocol kind of world. I think we're we're tending towards nation state no matter what MCP is trying to. do. I'd love for it to be, you know, more of a a a a regularized protocol. But, until then, you know, how flexible can you be within your cloud or whatever vendor vendor nation state or across the vendor nation states. And I think that's where partnerships and all that else stuff is also important. The. Well, last thing we should we should keep the. the question, yeah. is before. Yeah. you I think an interesting aspect of the stack flexibility that I I think is a little bit of a pullback is the question of, is this all gonna be in the cloud, or, you know, is there a resurgence sort of sort of hybrid and on prem? And we certainly have customers I've worked with multiple companies that have large, investments into their own GPU hardware to start to do. things like that. Hopefully, whether it's because of the questions of, you know, I need it because of secure data. I work in defense or I work you know, something like that. Or and this is, I think, becoming more common of, hey. We built this solution using, you know, a public API based AI tool, and it's really valuable for us. And we think it's you know, actually, this matters for us to do it, but at x sends a token, it we can't scale it to all of our users. Right? You know, we know this has ROI. We know this has benefit for us, but the ROI calculations don't work if I have to pay per token for 5,000, 10,000 people to use this, you know, tens, dozens, hundreds of times a day. And so people are really sort of starting to do that math of, hey. This might be $20,000,000 of hardware that I have to go build to or buy to go do this myself, but that's. six months of tokens for this one use case. And, okay, maybe. it's worth going and doing that. And so I do I'm I'm personally really curious because I have seen a little bit of a pullback from a sort of, like, cloud. I feel like for a while, it was almost like a cloud only mentality. And. the reality is that, obviously, every enterprise has legacy tech on prem, you know, Right. that's, like, impossible to it's not a possible escape, but it's very hard for us to sort of move out. But. I I actually think that it's not unlikely that as this space matures, that there will be a little bit more of a pullback to people being able to do more stuff on prem again because. of that those ROI calculations alongside, of course, the actual privacy and and, you know, sense The guardrails. Yeah. certain use cases. Yeah. So Yeah. Well, and that's interesting when you said that because it actually queues up the next decision at, that. I've, heard the slides that. look at that. Like, it's like you know what you're saying. You know, the multimodal reality. Right? So should I be all in on a multi LLM design, or do I need to accept constant switching as a permanent tax? And so 81% of these CIOs expect that their organization will reply rely on two or more LLM providers. to stay competitive. And nearly all CIOs agree that different LLMs perform better for different use cases and therefore require continual evaluation and switching. And so and it's interesting that, like, another, you know, a good half of these CIOs have already switched LLMs around, and an you know, another about 28% are saying they wanna consider it. But I think that goes exactly to your point of, like, sometimes it needs to be the cloud API model that works really well for certain use cases. And sometimes it's our highly secure private LLM that has been trained on our own internal dataset, and no one else can ever touch it. And we have a lot of, like, rails around it. And and it's not it's not a one size fit all. I think that's just I think that's just what that goes back to stack flexibility. Right? Like, not one vendor, not one LLM. You've got to have the ability to move these Lego pieces around and. rebuild. And the. the middle stat of the 93%, you know, sort of agree that different LMs will perform different sort of, like, there's a certain aspect of, like, well, duh. Right? Like, Duh. yes. Like, that's, like, that's what data scientists have built their the. last fifteen years is how do we figure out which right model is the right place for this. We're just sort of bringing more people to that party, I guess. Yes. But, Yeah. Yeah. Yeah. not to be snarky. Like, the thing that I also we really sort of have to acknowledge is that the the especially the PUP LLMs continue to change and evolve, and we sometimes see model degradation. We have seen issues and things where not even a new model came out that is better for it or whatever, but it's just like we can't even necessarily rely on the performance of any given model to stay static because they're not static. They're continually changing and being tuned in that. And I think another aspect of this is, you know, so a couple years ago, two two two and a half years ago, early stage stuff, lots of conversations with customers was around fine tuning LLMs. We're gonna take an. LLM and use our data and move to that, and we're gonna sort of to to tweak it with our and I was I was always a little skeptical when talking to those customers about that because I think that the knowledge and infrastructure and money and scale of data to really fine tune an LLM is was not out there for the most part. Yeah. Right? Yes. Enterprises have data if it was in the right shape, which is its own, question, but the skill sets weren't there. Right? Like, you know, Mhmm. this highly advanced NLP, LLM sort of, like, the the, like, actual skill sets in that level of AI and doing these, like, multibillion parameter models and stuff is there are very few people who have done that. There are still not a ton of people who've done that, but three years into this journey, there are more. Right? And so there are people now that are not just at these AI research companies that are working in industry moving around. And we've all had three years in the sort of the the data science community to learn more about and and do more with these things. And so that's only going to continue to accelerate. Right? You know, I think that there was questions a lot at that same time period of, is there gonna be, like, four or five foundation models? And that will be, you know, basically the whole space, or are there gonna be lots and lots and lots and lots of models? We. see we see they'll be moving down that lots and lots and lots of. models sort of path. Yeah. And so it's not just the multimality reality here. It is being able to rapidly evaluate and decide what is the best model today, tomorrow, next Monday, next year. And and who knows? Five years from now, maybe LLMs will already be dead five years from now. Who knows? We'll have moved on to the next phase of whatever Gen AI is. I. Yeah. always love. saying stuff on camera. Those words will come back to haunt me, I'm sure. This this is just like, the pace of change that is happening here, this the LLMs, a 100% to your point, like, nobody can say, yes. This is my my one. LLM that I use. I think that that was the writing was on the wall for that two, three years ago. Now it's. sort of like a conclusion. Well and and to that point then, because there is so much so many options out there, you then get this huge sprawl issue, Yes. which is sort of that this one of the the main main decisions for CIOs, which is how do I scale this AI stuff without creating shadow AI, data exposure, runaway sprawl? And, and 82% of CIOs agree employees are creating AI agents and apps faster than IT's ability to govern them. And 89% agree that broader employee access to AI tools without a strong governance program will create significant technical debt from ShadowAI. And and I think that, like, that's the that's the thing where it's everything is available and open and exciting, but then if you're going to do that, you have to be, able to create a safe place to experiment to try to grow before you then scale it out. And and I think this is actually interesting. There's a question here in the chat too about privacy concerns around, like, PII and and personal data. Is the public cloud secure for data? Well, I mean, private private data storage versus public data data storage has always been an area of debate. But we know that, like, there are ways to protect your data, your client data when it's in the cloud. Like, there are controls. There are cybersec controls. There are IT controls. There are data access controls that companies are using today. But the question then becomes, how do you govern the use of that data in a new sort of AI agentic world and do so in a way that is not going to create more, like, more shadow AI or more points of failure or data leakage or whatever it could be. And so the issue is not again, it's not about whether or not the data is secure because I think we've got a pretty good handle on that for the most part. But whether or not the people who are using that data to then drive decisions or make new AI systems, whether or not they're actually using it in the right way or in the way that it was intended. And that's where that governance piece becomes so critical, I think. I I I think I was actually just having a conversation with my wife weirdly this weekend. We were talking about, you know, like, the, like, data concerns with AI and all that sort of stuff. And, you know, the. failure points of data leakage, data, you know, breaches, stuff like that, nine times out of 10 are the people, not the systems. Right? And nine times out of 10 may be under saving it. Right? But I don't know that sort of to have the exact numbers. Right? But, like, Yeah. the the leakages come from people putting data that they're not supposed to put into it, things like that, far more than the systems and system accesses. And yes, building can guardrails that will identify somebody who's trying to put PII into, you know, this public one that they're not supposed to use, whatever, I think, is a challenge. And this is where I do you know, some of our customers very much come from the perspective of, hey. If it's touching this system, it's only gonna be our internal AI, period. Right? Right. Just because we. just don't we are not willing to accept that risk. Yes. We can put the right guardrails, this and that, whatever, but, like, there's only so much. People will find a way, deliberately or not, to get around your cartoon. To break things. They're. good out there. People. will break things. Yeah. And and then this think that also speaks to, education and, like, enablement in the right way too. It's not just like, here are the keys to your Ferrari, sir, but also here's how you need to use this thing responsibly, and here's what it means to use it responsibly and, like, why you shouldn't put client data into, you know, to Google, whatever it is. And so, I think I think there's, like, that that other aspect of it. This like, controlling the sprawls, not just about the technical, but also about the, like, motivational guardrails you put into place. And and it to come back to the sprawl piece a little bit too, you know, the the the sort of, like, data and privacy protections and things like that is one aspect of why this sprawl becomes a challenge. But the other piece that I do find really sort of, interesting in this, like, sprawl idea is that as much as, you know, people who have machine learning experience, data science experience, etcetera, you know, there's always been a talent gap there of somebody that could go in and sit down and clean data and train a model and for whatever use case. You know, the the reports haven't been, hey. We have all this capacity for production, and there's not enough data scientists to fill it. The reports for ten, fifteen years have been 60%, 80%, 90% of data science projects never make it to production. Right? So there was always a bottleneck on getting from a prototype or an early stage of a a building a project, a building at some sort of data product, and getting it to production. That's always been the bottleneck. Right? As much as. a scarcity of talent is a real thing, we were always building more than than we could get out for. And a lot of it came down to the lack of guardrails, the lack of trust things. There were other technical issues in it too, which AI is starting to help us move along. But the regulatory questions, the governance questions, the security and the privacy, and all of that sort of stuff, that was always a much harder aspect of working through getting something into production than necessarily the technical aspects of it. And all we're doing. is just widening the aperture on the stuff that comes in. Right? There's just now as far as the agent development, cloud code, all those sorts of things. And so this I like, I think the 82% question is kind of funny to me because, like, that is a 100% of every organization that's out there has more people that can build than can productionalize. And and more to. the point, Yeah. has the system enabled people to build, but not the systems built out to push things over to production. Yeah. It's not a new problem. It is just a problem that we are going to make exponentially. And this is a thing that I find when I'm talking to customers, especially when they're asking about, you know, agent building tools. You know? What, you know, we have to do to code agents, to build agents with our, you know, visual tools or or five coding or whatever it is. You know? Yes. I think it's important that we're building those tools. You know? I think that they'll also become commodity to some extent relatively soon. The ability to build and to code is, you know, becoming the low the barrier to the barrier to entry there has becomes very, very low and is continuing. Very small. Yeah. Code and good code is two different sorts of things and whatever. But the ability to to know if what you've built should go off to production and the systems to move those things into user hands, that's still the missing piece of the equation. And so, you know, Yep. it could quickly go from 60% of data science projects or AI projects don't see the light of day or 80% or whatever the, you know, most recent survey says to 99.9% of AI projects never. see the light of day. if we don't. fix piece of the problem. Yeah. Well, that's interesting because I know I know we're getting close to the end of the webinar, but ties in very nicely with what this this last decision making thing, decision making, you know, for CIOs is is can I prove that AI is adding business value before it gets treated like a cost center and cut like one? And so there is that ongoing question of, okay. Everybody's creating AI agents. We're getting all this AI sprawl, but what value are we getting out of it from sending, you know, 5,000 tokens out every day or whatever it is? And 98% of CIOs think that there's a lot of broad board pressure to demonstrate the measurable ROI from AI. And and 76% say that that pressure has actually risen quite significantly. And if they don't meet their budget, 71% or if they don't meet their targets, 71% of CIOs actually think their budget will be cut this year. And so it's not just so so what you're saying really resonates with that that decision for for CIOs. How do I figure out the ROI on a project, or on this AI sprawl that's getting built and prioritize the ones that I know are gonna give us back that value? And figuring out that proof point, that value point before it goes out to production is is gonna be important. And I think that in doing that activity, we can start getting more things into production versus being overwhelmed by the sprawl. Hey. I I had a really interesting, conversation last week. I was out at the Gartner conference in in Orlando, and I hosted a boardroom session with a couple dozen, CDOs and and CIOs. And we were talking about sort of the a framework for success. Right? And we sort of talked about, you know, like, level one AI success, level two, level three, sort of, like, you know, from the, like, use case to department level to sort of, like, enterprise level and just through thinking through that. But I did sort of an informal poll of the room sort of like, hey. Who's at the sort of, like, use case level success? Who's at sort of, know, like, larger scale success? And then who's really done it from an enterprise perspective? And, you know, I'd say maybe a quarter of the room sort of raised their hand about level one, and then maybe, like, a fifth or a sixth of the room raises their hand for level two. Nobody said they were at level three, but then the math obviously says that more than half of the people aren't even on the board. Right? And so we go back to this stuff that we're saying that where the 84%, 86%, 90%, whatever, of companies have AI in production. Right? We talked about 25% of people in this survey have AI in production in critical things. But when. you actually talk to companies about success and ROI for their AI investments, less than 10% concretely come back and say, yes. I have positive ROI on my AI. investment, especially my AI investments as a whole, let alone or, I guess, let alone being able to do it as a specific use case. I spent this money on this piece, and, yes, it's working. Or much have I spent in total to be able to do that? Mhmm. And this is, Mhmm. I think, the thing that it was easy to see that two years ago when every board again, I'm hyperbolizing. But, you know, boards were saying, hey. AI is is coming. You need to have a. strategy for this. You've got it. Like, Mhmm. never had I seen that sort of pressure from boards to technology and CEOs and and tech leaders to adopt a new technology that was, like, a year. So rapidly. Right? Yeah. That was that was intense. Yeah. Yeah. But everybody knew that it wasn't gonna be very long until those same boards came back and said, Mhmm. cool. I gave. you a big pile of money to go sign checks for name it. What have you now done with that money? Right? Like, Right. that is we think in terms of quarters, this has been a sort of you know, like, it hasn't been a, okay. I gave you this money. What did you do for me next quarter? But two years is a long time for boards who are looking at this quarter by quarter by. quarter, and. a lot of money has been spent. And, yes, there is absolutely ROI to be had here, but the proof points. and the ROI at the that companies need to justify the spend that they've done so far, it's not there yet for most. It's not there. Yeah. And and, actually, as as you were saying that, a very interesting question came into the chat, right, which is how do you ensure cross functional governance and alignment between the CFO, the COO, the CIO? And I think it's interesting because we we can think about, oh, there needs to be alignment across these big guys up at the guys. These big people up at the top. But in fact, it's it's whatever that alignment is, whatever that decision is, needs to be grounded in an actual business problem that only can be solved with AI or, like, can be better solved with AI. I think what happens is that we see these tools and we throw everything up the wall and see what sticks, and that's necessarily going to be that's not necessarily gonna create the misalignment versus having a set vision of, like, we have a specific business problem or we have a specific area of work that we need to think about and how we can optimize it for these goals and these ends, you know, before we start digging into the tech and the throwing and figuring out how it works. And that's I think that's a key way to actually start getting the alignment across the top top level. I totally agree. Like, a targeted business problem that you're trying to solve is. step one. Right? If, Right. like otherwise, Yeah. you don't ever wanna be regardless of if you're the CFO, the CIO, or the COO, a person with a, you know, a technology in search of a problem to solve. Right? Like, you. start out. from the problem side, and then this and then, you know, yes, it's important to have those you know, the the c suite people all agreed that this is the thing we're trying to solve. But down in the you know, at the various layers of people who are gonna be building and working with all of this, like, are they all aligned around this is a problem we're trying to solve, and here's where we're trying to go to do that. And. to sort of tie that into another question in the chat around from from Tommy around doing, like, do we do easier AI projects where the ROI is gonna be sort of quicker and more predictable, or do we do these sort of, like, you know, bigger, higher risk projects? I I will personally say, you know, after many years, we've sort of seen the the how you pick these projects, these business problems try to solve is actually a big differentiator between companies that are successful, the investments, and those that fail. And I think that the answer to that question is yes. You choose both of those things. You cannot just choose the quick wins because it's just not it doesn't it doesn't give the high level impact that you need. It's not changing. things enough. Right? It's I. often would joke about this, but it's a little more sort of, like, hands on, but, know, you'd say, this is the thing that we're gonna solve. We can do it in three weeks. Bada bing, bada boom. Man, alright. It's better. You now have a better spreadsheet than you used to. Right? And it's like, Yeah. you know, Yeah. Yeah. there's. better math in the spreadsheet or. something. It's like, you know, what we would sort of do ended up being so underwhelming. And it yes. It has ROI, but it's incremental ROI in these things. You can't just do those quick wins, but you also can't just say, hey. Thanks for my $20,000,000. We're gonna go do this lighthouse project. I'll see you in two. years. Right? Yeah. And this is what. I've seen. I've seen companies and and, you know, leaders do either one of these things where it's just every week is talking about what are we doing next week, and what are. we doing and, like, there is a real there is a real urgency to make sure that you are picking use cases that will show ROI quickly to justify the investment. But you have. to marry that with one or two hairier, larger scale, Bigger. large projects. Yeah. Yeah. I'd say I'd and one or two is probably so I'd say two or three is probably a better way to say it because those also have a higher risk. Right? It's so, like, you. marry the stuff that is, like, low risk and hopefully, like, low to medium reward, but, like, sort of those are the no brainers. Sure. Sometimes. you find to think, like, holy crap. This will be a piece of cake, and it's gonna make us a $100,000,000. Okay. Sure. Yeah. Every while something like that comes up. But, usually, you're sort of like, it's a high risk, high reward or low risk, low reward sort of situation, and you have to. pick some of those. Yeah. Have to have that sort of, like, balanced portfolio that is both. short term and long term. And long term. Yeah. Yeah. I know we're we're coming short on time. I I wanted to answer this question here from Thomas about, who owns the final decision at each stop or scale gate? Is it the CIO? Is it the CFO or joint governance? And how are conflicting priorities between compliance and, speed resolved from practice? And so I think from my perspective, what I've seen of governance at at my customers, right, is it is a it is a joint thing. Right? There is a, a decision making that happens together. And and I will say that it's not on every single use case. Right? Like, the CIO, the CFO, we're not sitting down and looking at, okay. Today, data scientist x is building this model in this notebook. What do we think of that? Yep. No. That that's not the kind of gating that we need, but it is the bigger question of what are our goals, what do we expect If we are planning to move forward with with this kind of, ambition, how do we keep it accountable? And that is like that that's the bigger AI COE, bigger sort of AI risk in tandem with cybersecurity, in tandem with business value, working together to drive that forward. And I think that the the sort of conflicting priorities you know, I think now the speed question is becoming less about we gotta get this out. Gotta get this out, but more about can we at least understand the value it's going to deliver and then build it within those compliance constraints versus we just need to get it out the door tomorrow. You know? But it's a but it's a it's a it's an ongoing push and pull. I don't think that there is a a single right answer for that, but it is gonna be very, you know, back to like, not back to back, but, like, give and take as you go through thing I say use case. about this idea of sort of, you know, the the stage gates are, you know, sort of how are you thinking about it is that you should have differential governance depending on the sort of, like, criticality of the project. Right? Or the risk of the project. Or the risk. Yeah. Yeah. Exactly. Right? Like, there's factors. in there that sort of shouldn't say in the same way that we think about, you know, sort of, like, medallion level data. Think about, Mhmm. you know, data projects in sort of, like, that same sort of way because something that I'm building that's for my team or for myself that I just, you know, want, you know, an agent to go and run a report for me every day or, you know, mean, hopefully, I'm not using this or something super pedestrian. But, like, you know, an agent is not gonna do something for me or for my team every like, okay. Cool. Have a couple checks. there to make sure it's not accessing data that it shouldn't that it's, you know, using role based access controls or something to just, you know, sort of, like, do that. But, like, that should not be a thing that I'm gonna have to go through some sort of giant governance process to do. And sometimes we we do that. We sort of say, well, okay. It's touching AI. So here's this humongous, you know, sort of, like, governance process that everything has to go through. And then all you do is create shadow. People will go around you to get access to things. versus. something that is touching customer PAI or PHI or, Mhmm. you know, for. things that, like, yes. There is risk here. There is or it's something this is customer facing. Of course, it needs to go through more governance. And so that's. sort of how, like, you know, players players come in and they have contextual governance. Absolutely. Yeah. Right? Yeah. Like, we have sort of the, like, here's the 12 steps that big projects go through, and then, you know, here's the, like, three of those that every project goes through. And you sort of you know, whether it's yeah. You have three different pathways or you just sort of have a quick does it have these things? Yes or no? Then cool. Now it only has to go through this. You have. to think about that because there isn't an answer of who is the, you know, right person for every stage gate. You know, there's. a there's not one person for all the stage gates. Right? I. think. an important thing that is does have to happen here is that there is only one person per stage gate. Like, Yeah. that's. all. like, you cannot have decision by consensus in those different places. You should absolutely have a place where. your chief privacy officer, Yeah. compliance team, or whoever can say no. And, you know, this because it's the compliance gate where it's the whatever. But at every single one, including the final sort of go, no, go of is this fine, you need to have one person that is clearly the person that makes that decision the sign off. Yeah. individual step of the way, but it also shouldn't be the. same person making that decision at every step of the Yeah. Hopefully, I said that. in. I I I am good. We have nine seconds. So check out the full report, at the the QR below, or find it on our website. Thank you so much, Connor. This was a lot of fun. was a lot of fun. It's been too long for. me. Thank you, I, know. for joining us. You know, go here to download the link. You can find both of us on LinkedIn. If there's questions we didn't get to that you wanna pick our brains, we are always more than happy to have these conversations. But thanks again for joining us this morning, this afternoon, wherever you happen to be, maybe this evening. Have a great day, everybody. Bye bye.