Video: Do More With Deep Agents | Duration: 2020s | Summary: Do More With Deep Agents | Chapters: Welcome and Introduction (7.04s), Deep Agent Advantages (233.965s), Deep Agent Demo (546.8s)
Transcript for "Do More With Deep Agents":
Hi, everyone. Welcome to our DeepAgents webinar. Just a few housekeeping things before we get started. This session will be recorded, and both the recording and slides will be sent to you via email after the webinar wraps. At the end of the presentation and demo, there will be a q and a section. So you can submit your questions at any time in the chat area. There's a q and a tab where you can add them, and we'll answer any questions during that q and a session. So I'll just give another minute here before we get started for any final guests to join. Okay. So please welcome our speakers today, Yousef and Chris. Yousef is a GenAI field specialist at Dataiku, and you may remember Chris if you attended our last webinar on agent orchestration. Before we jump into deep agents, I just wanted to give everyone a quick rundown of Dataiku for those of you who are not familiar. Dataiku is the universal AI platform that brings together your people, data, experts, and business unit users, your data, so in whatever format and wherever it lives, your governance for visibility and control across the organization covering all steps in the data to AI life cycle and your technology. So Dataiku is tech agnostic, which means you can swap out any cloud providers, data storage, or underlying models, and Dataiku works as a unifying layer on top of pretty much any tech stack. Concretely, Dataiku helps business users create everything from visualizations to Gen AI apps and agents right alongside code first users, like data scientists working in the same interface to build shared understanding and ensure teams can drive impact with data. At the same time, Dataiku makes sure that IT and compliance teams are able to keep track of what gets created, how data is being used, and enforce their unique quality, cost, and security requirements. As you may have experienced, enterprises are racing to adopt AI agents, and they're all kind of running into the same challenges. So scattered experiments, limited adoption, matching risk, and difficulty proving ROI. Employees are experimenting by experimenting by spinning up their own GPTs and siloed tools, while IT faces shadow AI and CIOs have a really unclear path to enterprise wide value. Despite that business despite massive investment, businesses are really struggling to actually see ROI and drive adoption engagement. So that's why we built AgentHub, which is a collaborative workspace in Dataiku's universal AI platform where any employee can find, use, and create trusted agents while IT maintains end to end control over the entire agent lifecycle. Today, Yousef and Chris are going to dive deep into the world of deep agents, so what that means for your business as well as demo just exactly what that looks like in Dataiku. So I'm going to hand that off to Yousef now. Thank you, Julia. So, hi, everyone. Just a quick reminder of the agenda today. We will start by, looking at why deep agent are the next crucial step for enterprise AI today. Then we will dive into some deep agents in practice use cases, and then we'll follow-up with the demo on the platform. And, we will leave some time for q and a at the end. So, as I said, deep agents are are becoming, are the necessary next step for, complex multi step workflows and enterprise AI. And, to start with that, let's let's start with the the actual, and current, agent implementations we're seeing today in most enterprises or what we call, shallow agents. So shallow agents are great for simple chat but may fail at complex enterprise workflows. As we see on the diagram, in the right, shallow agents basically are React agents capable or capable of interaction with their environment, but, basically, what happens behind the scene is the model gets an input from the user, then it could leverage some tools through actions and observation, and when it thinks that the task is done, it will output the final results to the user. So as I said, shallow agents are great for simple chats, but have have some issues, when it comes to tackling really long and complex asynchronous tasks. The limitation manifests in three failure modes for channel agents. First, it's the context saturation and poisoning, and this could be either through hitting the token limits, so basically retrieving some context that would truncate and hit the limit the the context limit or size of the LLM, so basically, triggering truncation. Or, even worse, they could suffer from the lost in the middle phenomenon where reasoning degrades as the volume of noise increases within the context. Second, shallow agents, are could also lose focus from attention drift. So basically, if they get into what we call a rabbit hole, where the agent tried to solve the wrong problem or trying to make the wrong tool call, it will get stuck and forget, forget the the lane goal the user is asking for. And last but not least, the lack of persistence. Let's let's say we have a complex 10 step process that fails at step nine. As the agent is stateless, it has no way to get back and restore from from that step. So, the agent cannot pivot its strategy based on on findings, on intermediary findings because it hasn't stored that durably into, into its state. This brings us to deep agents, autonomous systems designed for long horizon tasks. So how do deep agents overcome limitations of, of shallow agents? It is done through three main capabilities. First, through recursive planning or in simple terms, a to do list. Deep agents are capable of breaking down requests, so drafting a plan and also pivoting on that plan based on new data input. Second, delegation. Deep agents are capable of spawning sub agents for specific tasks, and this leads to, smart context loading. So both offloading and isolation as sub agents can be thought of as specialized agents on a on a given subject, and they will not be triggered, and they will be triggered by the the orchestrator, only when they are needed. And, finally, persistence. So using phase systems or or state to a fluid context, this could be done through, short term or long term memory for agents. So the orchestrator does not do the work here. Basically, it manages the work through memory, through sub agents, and using, context engineering. Let's move on to seeing deep agents in practice. So we have here a basic implementation on the left of shallow agents, which is basically an agent with, some tool calls. You can, identify as sub agents. And on the right, you have the deep agent implementation with additional, tools we have defined in our data cube project, which are which are file system and to do planning tools. Let's look now at a concrete, practical example within supply chain. So supply chain operations team are tasked with, ensuring supply chain resilience in order to migrate, mitigate operational risks like delays and stockouts. And for that, they need to, rapidly identify critical shipment delays across thousands of suppliers and active routes and determine, cross reference that with complex legal contracts to determine, liability. And so, deep agents could could help could help them both, orchestrate this multi step workflow for them, and second, have some actionable output like drafting emails to suppliers based on findings. Okay. Let's move to the exciting part of of this arena. So, I'll start sharing my screen to show you a quick demo of how the agents could be implemented in the table. K. Alright. Do you see my screen? Yeah. All good. Okay. Great. So I make a quick presentation of, Dataiku for those of you who don't know it. So, we're here on on the main, view on Dataiku, echo, which is the the flow view. So, basically, this is where, all people within your organization working on that echo, business people and data people should and would be collaborating in order to, to bring a use case to life from from data loading to preparation to AI and ML, stuff, and finally to to consumption through web applications, APIs, automations, or, whatever final consumption we have. I like to present this in terms of color. So, in simple terms, data would be in blue and agentic and generative AI blocks would be in pink. So for our use case, we have some input data, the some structured data about the logistics and the inventory and some unstructured data that could be, for example, contracts from suppliers, as we can see here, or communication templates that that could serve as guidelines for the model. So, data is in blue as is as I said and then, for the pink blocks we could have knowledge banks. So you can think of this as a smart search engine for agents to look for, to look into a structured data, such as contracts or, email templates. And we have we can also see the definition of our sub agents that the our main orchestrator could, could spawn later. So we have, analysts that go through structured data, so logistics analysts and inventory analysts. Then we have, contracts agent and the communication analyst agent the communication agent also that go through our contracts and emails. And at the end, the flow is very native and it's very simple to to get, what's happening here. So we have our orchestrator which is linked to our, sub agents, and, we also have the deep implementation, of this orchestrator where if I click on it, we could see that we have, additional tools we have added like the file system one and the to do planning. And this comes with some enhanced also system prompting in order to tell, the orchestrator when to use each tool and how. It also comes with a with a sandbox that serves as a file system for our agent to read, write, and search for files during execution. This this, as Julia presented it, this these agents could be consumed through agent hub, which is our solution for agentic chat application. So, if I look at, the shell agent implementation, and I I ask it for a task, usually what people tend to do, with these kinds of agents is try to try to really split the task into very, simple, subtasks. So basically doing the work, instead of the agents and trying to get it to to the right direction. So on this example, I would first ask you to identify all delayed shipments coming from from a given country, Vietnam here, which are critical. Then I would ask you to find, what's our penalty policy based on contracts we have for those suppliers. Then I would ask to check for inventory levels to see if there's, like, some very critical supplier, product missing. And then at the end, I would ask it to to draft the appropriate emails to suppliers to mitigate this situation. But if you think of this task, I'm sure many of you have already have already it's already the case for many of you. If you have a very long task, people usually do not mind having this task running for, maybe a longer time, asynchronously. And, but, but we, we are still needing that depth from the agent, in terms of, in terms of answer. So, what we could do with our deep agent implementation is put all that, the chat queries we had with our shallow agent into one one query, and let the agent do the work for, for half an hour or, like, for more time than the the shallow agent and then come back later to see, findings. And we could see here that the DIP agent could, implement and find, the key results we we were searching for. So it finds the delayed, shipments. It finds, our penalty, policy, the inventory risk, the strategy, then it drops some emails for us. And, also important, it saves some file to to its persistent file system as we have asked him, for in our query. So if we go to that file to that folder, sorry, we could see the to do list that the agent has drafted in order to tackle the task, and we could see that this is really complex multi step task needing spawning multiples of agents and maybe even having some shifts, in the in the to dos the agent has, has followed. You could also see some outputs from, intermediate steps like outputs from sub agents. And, most importantly, we could see the the image the agent has drafted for us. So I think this is really useful for these kinds of tasks as it's only one shot query to the patient. Also a very interesting feature of our latest version of Dataiku is the evaluation part. So in order to to see how how our deep implementation, behaves in terms of quality, latency, cost, compared to to the shallow agent implementation. We came up with a with a ground truth dataset, which has some user prompts, some category of use case, and then ground truth output we are expecting from our agents and then using some some deterministic metrics, for example, as as a build score or as as cost and latency, and then an LLM as a judge metrics, as as answer correctness and similarity, also agent goal accuracy. We used we used here GPT 4.1 minutei to evaluate our answers. Basically, what this recipe will do, it will take our ground truth datasets, meaning the user inputs, the expected outputs, then outputs from both our agents, and we'll evaluate them. And we could, this is really, really useful to have a comparison at the end of both. And no surprises here, we see that the deep agent, usually follows a more efficient path to to, to achieve the user's goal. It has better answer correctness, but it comes with, with a little bit of, of, uprising in terms of cost, which is expected. So, you can think of these agents are as more capable and more suited to complex tasks, but it came at the cost of pricing and also some latency sometimes. So this is the demo I hope that we that we presented to you how sorry. I'm back to the slides. Yeah. I hope that this demo has presented to you, most of the capabilities in terms of, GenAI, within our Dataiku platform. And I hope that this helps you, make make the choice of the right agent you could implement for your use case and, most importantly, how to evaluate it. So I think the next, next section would be q and a. So we would be happy, Yes. Chris and I, to answer your questions. Yeah, there are some questions already, in the chat. Some easy ones, so maybe we can get started with those. And feel free to add any questions, in the Q and A section and we'll go through these one by one here. So one of the ones, I think, was, like, which version of Dataiku are we using? This is 14.3. So, latest version of of Dataiku, that we're we're leveraging here. Let's go through some of these. So, Tom Brown, asked, is this for single runs or for single staff personal analyst, or is this for bulk review, of data? Lisa, do you wanna do you wanna take that one? So, like, what how would you how would you apply these these deep agents on what kind of tasks? I would say it it it could be applied to both. So, it's within the technical capabilities to have either single runs, or single person, as you said, or for bulk review of data. I think I think the this might not be the criteria for shallow versus deep, but it's more like the depth and complexity of the task we are trying to achieve. Is it something, requiring spawning multiple sub agents? Is it a long running task that needs the agent to have reasoning capabilities and drafting it to do to know, to not lose, not lose the focus, but both are doable. Yeah. I I would, I would completely agree that. I think, it really depends on task complexity. There's a and I would always say that's why the evaluations are so important. Right? If you're not sure of, like, is this really a complex task or not, running those evaluations side by side to understand, you know, what type of agentic implementation, works best from a cost perspective, but also from a performance perspective. I think that's exactly what the agent evaluation recipe is there for. So, yeah, for sure. Let me see here. Deep agents functionality is available in which version of Dataiku? So I'll I'll I'll take that one as well. So, the deep agent functionality that we've shown here is, essentially using Dataiku's visual agent functionality, which should be available in, in all the recent versions of Dataiku. I'll check for you exactly which which one that got released into, but, yeah, anything above, 13, should have that, for sure. What I would say is that the way we tackle this is by adding some custom tools to the agent toolkit to get it to perform these complex tasks. These tasks are like to do lists and context summarization, that is essentially allowing it to to behave like a deep agent. So, yeah, if that's the topic that you're interested in, please feel free to reach out, to, to either Yusuf or myself on on LinkedIn. We'd be more than happy to, to discuss it further on how you could implement this, in your in your real world use cases, in Dataiku. Let's see here. How do sub agents work in practice, and when should you delegate tests to them? Do sub agents have smaller models for context? Yousef, do you wanna take that one? Yeah. Sure. So thank you for the question, Julia. So sub agents in practice work are as completely, independent agents that the, orchestrator could, could spawn. And you could you can think of sub agents as, as tools, that the agent has access to. And, most important, these are, independent and skilled tools in terms of, of, I would say, specialized agents. So, to make it more practical, let's say we have our orchestrator, then we have maybe one research sub agent, one, communication sub agents. And these are really, within that they could define as as full capabilities agents. So you could use them even, on other, workflows or projects or or use cases as, as a full, cape fully capable, agent. The way I think what is powerful with our platform is that the the way we can leverage agents as other tools, to our main, agent. And, usually, yeah, sub agents might have smaller models for context. So, for this use case, this is what we have done. We think that, the main agent needed a very powerful model, so it was GPT five, because it needed some, reasoning capabilities and, long term planning. That sub agents were only, created with, more scoped and more, I would say straightforward tasks. So, more, smaller model and, smaller and faster model, have been deployed to to the to those, like. GPT five Mini, for example. Yeah. And then, we got another question from from Tom. So there's a couple of questions here. So Tom, when does 14.3 get deployed onto the Dataiku cloud instance? The the I believe there's a small or there's a slow rollout going on at the moment, for for upgrades. But feel free to to reach out to your, customer success manager or your, point of contact at Dataiku if, if you need, if you need an upgrade quicker. Also feel free to reach out to me with your space ID and I can look into that to see when that's scheduled. Please feel free to reach out. Let's see here. Please tell us more about the agent evaluation plugin. I could take that one as well, Tom. So, good question. So the agent evaluation is actually not a plugin. It's a it's a native recipe to Dataiku, got released in fourteen point three. It's really cool. It's essentially taking the LLM evaluation recipe to, to the next level, by implementing a lot of the agents evaluation techniques. So it's really powerful. As you could see, Yousef used this to, evaluate deep agents versus shallow agents and then compare the two against each other. That's out of the box functionality released in fourteen point three. Once the environment you're on upgrades that, you should have access to it. I recommend you look at the documentation. It's great documentation. See here. Another question. So maybe one for you, Yousef. So how does evaluation prevent hallucinations when using, the GPT models in Dataiku? It's Yes. This is a very good question, And I think, we are still, tricky within the agentic space, we are still doing our best to in order to. prevent hallucination, and it's always, best to put, better to put numbers on on such subjects. So, this could be done, basically, the evaluation recipe I showed, gives you the opportunity to define some custom metrics for your use case. So this could be done through custom traits, defining what what type of hallucination you you want your system to fight against and then having an LLM as a judge evaluating through the input, expected output, and even the output and the even the the the calls the agent have made, to sub agents and to other tools, run some evaluation on, on this metric that could be hallucination. And then we got another good one, probably best for you. So, can you assign different models to each agent? Is there a way to select the best model for the task as, like, an auto mode? Good question also. So, yeah, just to remind that that ECU is is agnostic in terms of, LMS provider. So, basically, you could bring whatever model you want from, from your provider, like cloud providers or even, host some local models, on your instance. And then you could, really, fine tune each not fine tune, but, like, have fine grain model selected for each of the sub agents agents that could be some some coding tasks, and you know that model a you have deployed is better at coding tests than model b. This is this is possible, completely possible within the technical. As we said, we did we did the split based on on within our use case, we did it based on larger versus smaller models on, on tasks that you could do. If you know that this LLM is better at coding than this LLM, then I will keep this LLM b for orchestration, but we use LLM a, which is better at coding for that particular task. Yeah. And what I would add to that here is that, kind of building out agents and leveraging LLMs is not an exact science. Right? There's a lot of, kind of experimentation and benchmarking to see what is working well for which task. You can see this all the time, out out there on, on x or YouTube if you're if you're kind of following the cycle. You can see that, loads of different LMS are best suited at at various different tasks. And so what I always encourage people to do is not just to choose any single model, but indeed benchmark these models, on real tests, using the agent evaluate recipe, the LLM evaluation recipe, against ground ground truth so that you can feel confident in the decision that you're making for, you know, your supervisory agent, but also for those sub agents, that they're performing the task in the in the way they should. Another question is how is Dataiku different than Salesforce Agentforce? And I think that's a really good question as well. So I'll take that one. So the way you can see Dataiku is is, essentially, it's a orchestration layer that lives on top of your data ecosystem. Your data ecosystem might, might live on premise. You might have Databricks. You might have Snowflake. You might have Azure. And and Datadogu provides you an interface on top of that ecosystem to run very efficient, kind of data pipelines, and eventually into agentic pipelines as well. And so what I would say there is that, Datadogu is very different than than kind of Salesforce and and where we've come from. We come from really a pure data science world, and Salesforce obviously coming from from their world of of CRM and, working very, very closely with that type of data. And so really, very two very different products, but functionally, I would say that you could use both. So, if you wanted to leverage, certain agents, in Salesforce to perform certain tasks and and have those as sub agents in Dataiku, that's something that you could set up either through the MCP connection or, or via HTTP, of course, using the inline Python tool. So you can really make the decisions on which agents you wanna use, for which type of tasks. So, again, it goes back to being able to benchmark tasks against each each other to see what is the right tool, for for your problem. And I think we can probably do one more here. Let's see. I think we've done most. Any other questions someone would add here? Templates. That's a that's a good question. So, Tom, I think that was one of your first questions. So templates, yes, they actually do come out of the box with Agent Hub. So if you've installed Agent Hub, especially the latest version, you should be able to see a lot of the prompt templates that, that our SMEs have built. So we've got SMEs, in in all types of industries, and we've worked with them to build out, let's say, golden prompts, what great prompts look like. And I think that's a key takeaway that Yousafzai and I had when we were working with DeepAgents, is how impactful the prompts actually are. I always knew they were impactful, but really seeing how impactful they were was was quite even surprising to me. So I would say that, if you're gonna put in work when you're building out agentic tasks in in anything, it's, it's in building the the prompts out, and testing different prompts against each other. So, yeah, in agent hub, you can see it under the prompt library. I can send you the link for it, for it after as well. Oh, and I think that's most questions here. So, happy to wrap up there. Hope that was useful, for everyone. Thank you so much, Yousef, for for doing this work, doing doing this research. I know it's, it's a really cool cool subject and, certainly fascinate me. So, yeah, thanks very much for for joining, and, we hope to see you in the next one.