Video: AI Agents in Action: Accelerate Clinical Trials | Duration: 3300s | Summary: AI Agents in Action: Accelerate Clinical Trials | Chapters: Introduction to Webinar (5.44s), Agents as Amplifiers (174.91s), Understanding AI Agents (303.275s), Pharmaceutical AI Applications (534.89s), Clinical Trial Intelligence Agent (614.525s), Agent System Demonstration (1035.795s), Specialized Agent Orchestration (1986.725s), Agent Applications Explored (2120.535s), Closing and Q&A (2235.21s), Closing and Q&A (2437.655s)
Transcript for "AI Agents in Action: Accelerate Clinical Trials":
Yeah. Hello, everyone. I can see from the chat that there were some people who were quite impatient to have us get started. So, hopefully, now you're able to hear me. My name is Sophie. I'm based in France. Very pleased to have the opportunity to, be one of, your, co anchors today for this webinar. Today, we're gonna be speaking about AI agents, but more specifically, we're gonna be looking around how AI agents can deliver tangible business value in the pharmaceutical industries by focusing on one very specific space, which is clinical trials. So there's not just gonna be me who is gonna be speaking on this webinar, and I'm gonna let my colleague Liheng actually introduce himself. Liheng, take the leap and, tell us a little bit more about yourself. Yes. Thank you, Sophie. Hi, everybody. It's an honor to be here today, to talk about our urgent use case in clinical operations. I'm Li Hong. I'm a senior solution data scientist, for life sciences and health care at Dataiku. Nice to meet you all. So what we're gonna be covering today is gonna be the following. On my side, as I said, I was based in Paris. I'm the, senior vice president of products, for Dataiku, and we're gonna get started with looking at what exactly we're talking about when we're talking about First Aid IQ and about the agentic opportunity. Then we're gonna deep dive deeper into, practically speaking, what is it when we talk about AI agents for clinical operations. We will then do a demo, do a quick wrap up, and finish with a q and a. If you have questions as we go through, don't hesitate to put them in the chat and to put them in the q and a section. We're gonna be answering them at the end. But, you know, feel free to post them so as not to, forget about them. So let me get started with the presentation. Dataiku is a bit more than ten years old, and, as part of the work that we do, we happen to work with a bit more than 700 customers. Out of these customers, more than 50 of them are in the life sciences industry. So in life sciences, we will find customers that are in the pharmaceutical space, but also in the health care space, including some that are the global leading players, of these companies. Why this is important is that working with these different organizations have given us a privileged opportunity to see how they're using data, how you are using AI in different spaces, And with the arrival of the agentic opportunity, how this agentic opportunity materializes itself in different ways in different parts of the operational setup of these companies? Sorry for that. It seems like a little bug, for, more of my slides. Let me just un refresh. Share. Okay. Apparently, I have do have a little bug. I think we're gonna be skipping the, my data queue slide and moving yes. Thanks a lot for this. And apologize for that. Sometimes Wi Fi doesn't work as well as it should in different parts of the country. So the the reason why we're gonna be talking about agents today is that, first, of course, it's one of the most important, technological, innovation that has been happening in the market. But also because from our standpoint, we see agents as actually almost a magnifying glass of and the amplifier of all the investments that have been done on data, on analytics, and on models in the past. As you will see through the demonstration that we're gonna be making, to make effective agents, it is absolutely critical to have the opportunity to tie them to what represents the nature of your business, the nature of your organization. And if we take a step back and we look at how usually an organization is shaped, it is nicely represented by its data structure, by the pieces and analytics that have been highlighted to identify specific KPIs or specific indicators of activities, as well as by all the different business systems that can exist around. At Dataiku, our conviction is really that bringing all these different components together is what's gonna be critical in order to materialize that agentic value prop. Hence, why you can see on this slide that we have this continuity and also the deep routing to other types of business systems that you can see on this slide. So we're not here to talk just about data. When we take a step back, first things first, it's important to start by doing a definition around what an AI agent is about. There are many, many different ways of defining this, but at a very high level, an AI agent is an element powered system, meaning a system that will be leveraging one to many different LMS and different Geni technologies that is here to achieve different types of objectives through multiple steps. It can have multiple type of tools. It can have it can engage in multiple type of activities, and it goes beyond just the simple concept of retrieval. What we mean by retrieval is, of course, what you would find with an augmented LLM and an LLM to which you will have added your own company information as part of a knowledge bank. For agents, knowledge banks can be one of the sources of information, but it's only one of them or one type of tools that can be found with agents. So more practically, if we talk about the different components you will find in agents, an agent will be relying on one to many different elements to one to many different types of prompts, to different types of tools, which themselves will have their own descriptions. Tool can be the capacity to call a given data set, to call a given API, to perform different types of actions directly in a business system or via new protocols like MCP, or to query a given model to make a query on Internet, etcetera, etcetera. An agent can be on a stand alone as you can see in my slide or it can be a chain of different agents that actually themselves need to be orchestrated as almost sub tools to a macro agent acting as an orchestrator. At the end of the day, an agent itself only becomes a business system that needs to be queried and interacted with in different ways. And agents can be materialized can actually work in prompt, in prompt. In batch, meaning that every single day, you could have an agents being run to to to be applied to a new series of events, a new set of data. It can also, of course, be interactive live through direct user queries using a chat type of interface. This is all what we're gonna be seeing today. Agents can serve different types of purposes within a business organization. They can be extremely powerful worker augmentation tools, And this is what we're gonna be seeing with clinical trials. In clinical trials, we have people who need to make decisions around the best type of clinical site. In order to take the decision, that person will need to digest information coming from a number of different sources. That worker, that person that needs to take that decision can have his decision making process facilitated and augmented if the right type of information is pulled together itself by the agents. Agents also have the opportunity to to to support process automation. Process automation, we're really talking about looking at a process that starts from point a to point b, and an agent can act as this orchestrator that operates this entire value chain. And then at the end of the day, we can talk also about more sophisticated business revamping, like intelligence business chains being pulled together. Ultimately, this is what will allow organizations both to automate, to operationalize, to bring operational efficiency, but also to revamp their day to day activities. In pharma, there are many different examples. On the process automation side, pipeline generation, if we're talking, for instance, about, sales activities, but also about identifying more quickly candidates for different r and d steps, can be one of these examples of these process automation. Worker augmentation, we talked about it. We were really looking at people who have the type of skill set that is key for decision making in one of these organizations that usually themselves own very sophisticated and deep knowledge, but who have a decision making process that can take some time due to the complexity of the research they have to process. And this is where we believe that we see that agents have offer a significant acceleration potential. Mr. Diddy, one of our customers, has been very successful in accelerating the work of their patent attorneys with one of these, one of their agent at work. And last but not least, there's actually an opportunity to revamp complete end to end value chains, complete end to end processes where different people have to interact at different stages. This is exactly what we're gonna be looking at for clinical trials intelligence. Yang, do you wanna take the lead on this and take us through what clinical trials intelligence is about? Yes. Thank you, Sophie. So, today, I would like to introduce the agent system, clinical trial intelligence agent, which dynamically queries and synthesize information from new people sources to support clinical operations team in making informed decisions, especially around clinical site selection and investigation patient identification. I will highlight the core features of DataEase platform, Agentix, and generative AI capabilities, and this proposed frameworks aims to help our users jump start your custom agenting system using our platform. So back to this topic on clinical site planning, various data sources are needed to identify suitable sites and principal investigators for a given study protocol. For example, global trial registries like clinicaltrial.gov can identify similar studies and competitive landscapes. The CMS open payment database can map financial relationships between principal investigators, trials, and industry sponsors. The proprietary model can predict enrollment rates based on study design, and site mattresses from a CTMS system can demonstrate historical performance. The life sciences industry is increasingly adopting advanced data driven approaches, combining public and proprietary data to streamline decision making. A common challenge is the costly and time consuming process of manually collecting and integrating data from disparate systems. Without adequate IT and analytics analytics support, this data remains silos and inaccessible to domain experts and business users. Many cleanups teams retain their manual site screening on the candidates provided by third party CIOs, often missing enrollment windows while competitors secure the best sites and investigators. Instead of having IT and analytics teams manual collect manually collect data from heterogeneous sources, perform perform data wrangling, and precompute for every single use case. We believe that a agenda system allows business users to directly interact with their data using literal language and receive actionable insights on demand. So through a simple conversational interface supported by Dataiku agent Agent Connect. We will interact we will interact with the agent to search and compare clinical studies, prioritize clinical sites based on specific criteria, identify leading principal investigators, generate a competitiveness report for a selected researcher in the context of its target study. And finally, we will ask the agent to generate and send an email draft for further review, showcasing how these systems can integrate directly into decision making workflows. The clinical trial intelligence agent consists of an orchestrator and a network of agents powered by large language models and versatile agent tools. It contains a powerful semantic model that supports SQL queries on multiple datasets from two Dataiku's business solutions. Data ecosystems solutions are end to end data solutions targeting well scoped business needs and user stories available to all data users. The clinical trial intelligence agent leverages insights from the clinical sites intelligence solutions and social determinants help solution. The agent can also access two public databases via API calls. For this demo, we curated the clinical site intelligence solutions with 20,000 cancer studies from clinicaltrial.gov. Okay. Now let's, deep dive into this agent. This is showing. Alright. So so this is, the agent top next on data in this platform. This is where, allow users host the system and it's the back end of this web app agent connect. On the very top, it allows the users to choose a large language model of the choice available from your IT infrastructures to power this orchestrator. And down here, there's this place where it allows you to select multiple agent system from different projects, through, the platform. This, is a very, in is a very, useful features that allows the users to have access to different agent system across different projects without the need to copy over, for, every single use case. And in this demo, for this agentic system, we have two, major agents that power this system. The first agent is study information retrieval agents, which have multiple tools to make, database queries and API calls to get information for study protocols and clinical sites and principal investigators. The second agent is the actionable part, which can generate competitiveness support of a principal investigators for a given study, and it also allows the users to send out the draft to a designated email address. Alright. Let's get started. This is the front end of agent connect. It's like a a chatbot interface. And let's assume we have an active study, and we want to find studies similar to it. Pay attention to this little icon here. So the orchestrator send out the request to the agents that he has directly direct access to, and it will identify the correct sequence according to the prompt that you assigned to this orchestrator and then do the, following, task to achieve its goal. Here, it takes a channel thought approach to break down a more complicated task like this into smaller tangible steps and then find the, most of the digital tool to solve those tasks step by step. First, you look up, study by the study ID, which proves the details of the study protocols, and then use those protocols to make a semantic search against a vector stores to find the most similar studies from those two, 20,000, cancer studies that we've embedded for this demo. And once we have all the relevant informations, the orchestrator synthesize the informations. First, you listed the study protocols for study. And then, finally, it, synthesized its results the semantic search results and at least the most, relevant study from the vector space. And it's according, to how you design the prompt. You can list out the result. And then here, we ask, the agents always to provide the URL to the official website, to to the clinicaltrial.gov for as a reference of truth. Right. Now we have similar studies. Let's ask the agents to compare a given study against ours. Now while waiting for the current result, let's, take a deeper look into the workflow, the pipeline that powers this agent system. So this is the, project that holds this agentic system. And in the flow, this is where, the the project that visualize the data pipeline that, power this adjusted system. So for the audience who don't know, not who are not very familiar with the platform, a flow has two, basic components. The first component is dataset, which is where the data is from. And the second component is the recipe. So the recipe is from the basic data wrangling, like join groupings, PVA sorting, or it also allows more advanced users like, professional coders who might more customize, the functions. And there are also, recipes allows you to create machine learning recipes as well as to leverage the powerful, generative AI capabilities. And today, we will, highlights the embed datasets. This is how we can create the vector stores or in that data, we call them knowledge bank to, support the rack, functions. And the diamond recipe here, it's the agentic functions that we support, on Dataiku platform. So here, as we I have explained it earlier, we have leveraged, the info the insights that generated by two Dataiku solutions. So the black datasets are the insights that we have, reported from the Dataiku solutions, clinical site intelligence solutions, and the, social determinants of health solutions. And with proper, labeling, the which first create the data data, metadata to describe the datasets, and it's, each column. This is a very important steps, which will allow, our agents to understand where to identify, where to locate the data and then to make a SQL queries. Alright. Let's go back to the part search results. Here he returns. And then the the agents successfully queried the right study for patients and made an head to head head to head comparisons, categories by categories. And now we will ask you to list the, US clinical sites of this two this two studies. Let's continue to see, understand how we can create eight the, leverage the generating AI recipes to create a vector store that in that can embed the 20,000 cancer study protocols. So here, this embed recipe allows, you to, embed a lot of structured data with this metadata into a vector store. And here again, our platform is diagnostic, so it allows you to choose the embedded model that's available in bio IT, infrastructures as well as the vector store storage type and to embed, the documentations of your choice. And let's take a look how we can construct the, agents, the visual agents, the data API. So it's in the, generated AI tab functionalities. And then here, it is the step to create the agent. So Dataiku has, worked closely with our technology partner like Snowflake, Databricks, and Azure's. So we have the, we allow the users to leverage their, our technology technology partners' powerful, generated for a generic AI tools on our platform, as you can see here. The Snowflake codex agents or the Databricks genetic AI capabilities. Let's take a look at the studied information retrieval agents that powered these, the genetic system. Yeah. And you can choose the large language model of your choice. The prompt defines the role and the task, the goals it is, supposed to perform. And the, what distinguish an agent from a simple foundational model is that it has, the agentic tools which allows it to perform much different types of performance. In this, you start information retrieval by agent. You can use the agent tools to query multiple datasets to query the semantic, to perform a semantic search against the vector store or to make API calls. And on this interface, it allows our users to iteratively, build up the prompt and its tools with the quick test. And here you can see the response here instantly. Alright. So, here we can see the result from the questions that we just asked about the clinical science. These two studies He made a more complicated, SQL queries, multiple datasets, and nicely simplifies the results here. Because of the time issue, I just want to show, here, what what is the best questions I want to demo directly. In order to, encourage diverse recruitment, here we ask the agents to query against the social determinants of health informations and then to prioritize the clinical sites located in, in underserved communities for a given study. So the agent was able to make the joint between the study sites informations and the social determinants of health informations, And then it ranked the clinical sites by the social social vulnerability index of the county where the clinical site, is located. And then he also highlights the, prominent or leading socioeconomic factors for each clinical site. And, finally, I want to highlight the, how to create agent tools for the powerful agents that we just see. So here, the data use platform has reviewed a set of, agent tools for our users to use without writing a single code. So according to, OpenAI's, documentations, we have three types of agent tools. So first type is to make queries to retrieve information. For example, to look up a data set, to write, data SQL queries, to make semantic search against the vector store, to echo echo, so on and so forth. The second type of the agent tool is to create an output. Also, write down the output, for example, to write down or overwrite the datasets, to send out a message, or to push down the ticket to create the tickets, for example. And the third type is to leverage the other large language models or an other agent, SOQ. As you can see here, we these functions, these agent tool allows users to leverage other foundational models for agent as a tool. So finally, I would like to, show a, the competitive report that's generated that was generated by this, agent system. Can you can everybody see the report just to make sure that I'm showing the right screen? Nope. We're still seeing, your list of, agent. You're we're still seeing the list of tools. Yep. Let me choose to write this window. It's this one. Can everybody see it? It's a it's a PDF. We can't see your screen, Lian. Right now, we at at least I can't. Excuse me. What about now? Yes. Oh, yeah. I think it is. Yes. So to complement what our meeting was was showing before is, what's very interesting with, this type of agent is that we see that there are, a number of different ways of interacting with it. One way is indeed to have more of the conversational approach that Lian was showing before, where we have this orchestrator agent that is tapping into these multiple, specialized agents in order to provide the right type of answer based on the input and the chain of conversation. But here in that case, we actually have been using the power of all these different agents to produce an actual competitive report that can be used for further decision making. Is this, correct assessment, Yiding? Yes. So thank you, Sophie. So, yeah, lastly, of this demo, I'm showing the actionable insights generated by the clinical trial intelligence agent. This agent drafted a competitiveness report by gathering study and funding informations from various sources as you can see here. And this, it synthesized the data to, to recommend the principal investigators for a specific study, compiling this information into a customized report and distributing it via email for further review. Oh, thank you, everybody. Thank you so much. This is the end of the live demo. Sophie? Yeah. So we we're gonna go back to a few slides before taking the q and a. So, thanks about leaning for this. So I hope it gives you a good idea of how, such an agent can be put together. If we, and, Mariceth team, if you can kindly share the slides because my computer definitely doesn't wanna share mine. Thanks so much. So, if we if we can start back from, slide 15, that would be perfect. Thanks so much. So, yeah, if we take a step back on what Lehan was presenting just before, and this is what was illustrated on this slide here, we have seen that we have, an LMM that is used, in order to, tap into one study intelligence agent. That study intelligence agent itself is an orchestrator of different types of specialized agents that are either using data or using knowledge banks or using API calls in order to answer, the questions that Kalihan has been asking. You can see also here on this slide that, for some of them, we're actually calling, what we call on our side solutions. So solutions are actually fully fledged use case projects that are available within the DataIT platform and that are here as accelerators. In that case, we use one that is focused on SDLH and the other one that is focused on clinical site intelligence. It also just shows the potential of recomposing these different types of agents because what this study intelligence agent is doing today, tapping into SDOH potentially, that SDOH project could actually just be a, that SDOH agent could just be a powerful tool for another specialized agent working typically more towards recommendation of the right type of programs for, geographical allocation of programs for a health care product provider. And we also see here that there's a second agent that is specialized on that reports production that is using some of the same types of inputs, but also some different ones with a prompt that actually provides it, with the, the right type of framework that it's supposed to be following. This is just one example of one of the types of agents that can be powered, inside of an organization. As I was saying, one of the things that is important is that ultimately agents are just orchestrators, and they're orchestrators of capabilities. These capabilities can be composed in very different ways. Agents can be applied, of course, to that clinical site, clinical trial site selection process that we have been talking about, which is a key process in the life sciences value chain and one on which today there can be a lot of acceleration to be done and lot of more efficiency gains to be delivered. Also an opportunity for pharmaceutical companies to gain more autonomy versus CROs. But it's also some the same type of approach could be applied to pharmacovigilance typically to accelerate the capacity to surface the right type of alerts, but also to produce the right type of reports based on the initial alerts being generated. End to end drug launch management, so with the capacity to have, to support the different types of operationals, identifying what are the right steps to go towards in order to move, drug launch management from stage a to stage b. And last but not least, of course, everything that will be tied to personalized, health care provider engagements whereby, agents can actually support the sellers themselves in having access to the right information, understanding what is the right type of next best action with their different targets for their different, products, but also fully orchestrating that process with actual autonomous engagements, of the targets. So these are just some examples that, we are seeing out there and that are available through the usage of these types of agentic technologies. Now in order to close the session and to go into q and a, one of the things that I wanted to insist on is that ultimately with what you have seen today with Li Hue, we have been talking about agent building. So agent building will be about identifying the right type of sources, identifying the right type of structure, defining your prompt strategy, testing all these different types of prompts. But as we work on an agent building, we're actually just at one it's it's just one moment inside the AI agent value chain. Indeed, before that, there will be gonna be some questions around selecting the right type of elements, selecting the right type of third party agents, the right type of self built agents, the right type of tools, and connecting all of this together. Once the agent has been built, there will be a lot around testing, doing more quality assurance. So what Lihang illustrated to you today is the capacity to do some testing as building was being done, but there's much more that can be done in data accurate in order to be in investigating traces, in going deeper into LLM evaluation in order to have a spot analysis, but also running monitoring of the effectiveness of the agentic system. We also have in Dataiku the capacity to run inspections, controls, and sign up with what we call Dataiku governed because, ultimately, what Liheng has been doing on his side if he has he was working still in one of these pharmaceutical companies, would have been to be building four different people to use, and that would have been real requiring the right type of sign off from themselves, but also from someone who understands deeply the type of Gen AI risk we're talking about. And all of this would have been, required to go into continuous monitoring. And and this is gonna be a good translation to the q and a. I've seen that there were some questions around how do you select the right type of LMS? How do you identify what's a suitable LMS to work with? In data, you have the possibility to bring all the different types of elements that you decide to work with. These elements can be externally hosted elements like OpenAI, like the ones from Bedrock, but also can be self hosted ones that you can decide to have on your own infrastructure. We believe that actually it's not a one size fits all. Depending upon the sensitivity of your data, depending upon the the business targets that you have, you're gonna have to adjust your LMM strategies to these specificities. And I saw that there were also some questions around PII, for instance. Is there a way to identify if there's personal information that is passed at the LLM level? For that, there's the possibility, and they'd like you to put guardrails in place. But there's also gonna be some questions around what LLM and what LLM providers you trust to complete different tasks. And so it means that as part of this, there's gonna be one piece of your generalized strategy that will be around LLM suitability assessment and activation. Hence, all this first phase that you see here where we also provide the full continuity of capacity to both identify what are these providers that you decide to work with, do model due diligence, do actual testing using the capabilities, embedded in Dataiku, and then enforce, the the usage rules and guidelines before, integrating them into agent development, which is all that you find today in Dataiku as a comprehensive platform. So we're gonna close the session today on that. I hope it was interesting for you to see an actual agent in motion, and, we're gonna start taking some questions. I'm gonna go through the questions with Lihang that are in the q and a. If you have some more, don't hesitate to post them as we go through the questions. So, there was one the first questions that was asked was around, an architecture with multiple agents, but indeed, this one didn't show how, the these agents can iterate among themselves. That is something that actually can be done, back and forth between agents. It would not be done with the type of strategy that Liang used on his side. Liang used an a visual agent, to as the orchestrator, but it's definitely something that can be achieved with code agents. We're also looking at making sure that going forward, these, more advanced capabilities, stop loss, human in the loop interactions, and all of this, giving providing more history to the context will be available, as capabilities for visual agents. All, I so I was trying to answer the question around the basis of the LLM and how the LLMs have been selected, for these types of, tasks. We didn't go into this, but, Leheng, under your control, inside your whole present inside the whole agent that you the whole agentic structure you have presented, there's not just one LLM that is being used, but several. Right? Right. Exactly. And it is a it's a really process. So it's there's not really a consensus that what LN should you choose for specific task is more like a trial and error process during the development to to and and use, sometimes your self defined mattresses to evaluate the performance of a given, LM to a given task. So is it try and error irritating process to choose the most suitable, foundational model to use cases? What we have also, that we didn't so if we take, for instance, the example of agent connect. So agent connect is a front end that we provide to our customers, to allow, the to allow them to have access to a chat interface out of the box that can compose with, different agents. Within that type of, environment, typically, there's the different possibility to differ to use multiple elements for multiple types of tasks. If we're talking just about information summarization, there will be no need to use complex LMM that will have different types of cost. Whereas if we're talking about actual querying of a knowledge bank for which we need to have specific type of feedback, then it could be needed to use, an element that has a bit more power, just to give you some example. So that was around, element selection. One of the question that we got was around how does a user gauge whether or not the model is experimenting hallucination, which I think is a good question and not always an easy one to answer. Lian, do you wanna take the do you wanna take it give it a shot? Yeah. So in this demo, we this Reddit, we we proposed a rather relatively simple, framework. And at this stage, we didn't, we haven't put in, like, a validation, like, a layer on top of it to validate whether, the agenda system is representing the information truthfully. So it's, this type of, orchestrator and structure that definitely allows you to support you to build additional validation layer, which is like a validation agent to verify the result that each time or selectively maybe once every few times to validate, the result of, an agent. One of the things, again, that we didn't showcase today in this presentation is, the capacity to actually put after that more systematic evaluation in place. So once, as part of the work that would need to be done before going into production would be the need to have an SME in the loop type of validation to establish a golden dataset of expected behaviors. Once, that golden dataset is established using LLM eval, there is the possibility to actually put in place running control of the behavior of, the agentic system versus that l m evaluation. We're also working actually on extending those capabilities to more specific agent evaluation, which will be coming out in, in in in a couple of months. One one question that was asked is, I am new to agents. Can I envision an agent to be an intelligent background task, that has other advanced features like interacting with other agents and has seen its its own capabilities with time, etcetera, etcetera? I would say yes and no. The even when we talk about hallucinations, it's quite interesting to take a step back around what LLMs do. Ultimately, LLMs react to words. And so they're only as smart as the tools that we give to them, and they will always try to perform a task based on the instructions that they're being that that they receive. And so ultimately, even if they are, one of the big differences that we see today between machine learning models and the potential of Gen AI, it's true that machine learning models are much more point use cases. They're point solutions to answering very specific type of topics. Typically, for instance, if we're talking about, a scoring of clinical trial sites for, the for different types of needs, a machine learning model that would be doing this would be performing that specific task. And that specific task itself would become a tool for an agent. An agent cannot go beyond the tools that have been defined for itself. One of the beauties though of being in a type of environment that allows for composability, like the ones that we have in Dataiku, is that once you have done the first iteration of your agents and delivered first value, it's very easy to add to it another tool and yet another tool or have it communicate to another agent, etcetera, etcetera. Now before going in production, we would always advise you careful testing and backtesting of all of this, again, using, the capabilities that we're provide providing. Other question that we got was that the more elements involved, the more likelihood for hallucination and how to make it mitigate this. I would say yes and no on the likelihood for hallucination. It is true that today, the more you pile up LMs on top of LMs, the more it can become blurry. So it's it's a question of being quite specific in how we and and how how much autonomy we give to, to an LMM or not. As I was do going through my introduction, I explained that we have one of our customers today that is very successful in, augmenting the way that patent attorneys work with GenAI. The prompts that they have put together that is actually a system of change prompts represents 60 pages of prompts. And with that, I'm not gonna tell you that there's no hallucination risk. There's always some. Because, again, what an LLM will do is it will react to words, and words always have a specific meaning behind, and it will react to, to, composition of words together. But the fact that they that they have gone very thorough in defining their guardrails, their prompts, and all the tests that support it, make sure that today, they feel confident in that big pharma company, to actually trust their JI system in order to answer and to provide support to legal advisory, which is a good demonstration that even by combining elements together with the right type of prompting and the right type of guardrails, you can get that. Lian, question for you was, what you were doing was a mix of low code and no code. Right? It would also have, like this agent also includes some code agents, which is poorly composed of Python codes. So it's a mixture of everything from, like, high code to no codes. Which is something that actually we we believe a lot into. We we do believe that they like you a lot in the power of no of of no code because, a, we think that it supports significant acceleration of delivery. Second, that it lowers the barrier to, to delivery by broader, skill set and also lowers the barrier to collaboration. But sticking to no code only is actually also would be, would be a blocker to achieving specific types of business outcomes, hence the mix that was presenting today. There are also some moments where just doing a few lines of codes goes much faster than trying to use visual interfaces or where visual interfaces are not able to answer, to address, these specific types of notes. I think I've gone through actually most of the questions that have been asked on the queue on the chats. If there's any other ones, don't hesitate to put them before we wrap up with our webinar. There are a lot of questions of whether the material is gonna be public, whether it's gonna be possible to watch it in replay, whether it's gonna be possible to address, to access the presentation. There will be a follow-up emails that will be sent. In terms of making all of this public, today we're working hard on making sure that all the work we have been doing on our, our agents, actually, we turn them into solutions that will be fully embedded into Dataiku and, accessible to our customers, to offer these, safe, accelerators and starting points. More to come down this before the end of the year. Yeah. There are a few questions, like, new viewer questions. It's regarding the datasets it's using by this agent, and then what is the technology that generated AI, technology behind this agent. So, in short, we are only using the public available datasets to support to develop this agent system. So clinicaltrial.gov is one. The CMS open payment datasets is the other, and we also included The US census data and the CDC, disease information for for these purposes. But, this, this demo, want to emphasize that, it is, it will facilitate your user greatly to include datasets from various sources, including the public datasets and your proprietary data to support or to construct such a dedicated agenda system. And in terms of the technology, the GenAI capability, it's some we don't find we didn't find you the data. It's, we heavily we heavily rely on prompting, prompt engineering to give a clear role and description of the task that each agent should use and how, like, each agent tool should be used to perform, their designated roles in this system. And, yes, of course, we also include a rack that, allows the agentic system to make semantic search, to identify similar studies. I think we have, covered most of the questions. Yes. I think we have. So thanks a lot, everyone, and thanks a lot for the questions. The types of questions you were asking, I'm sure that you really have been staying with us and and listening to, to the the presentation we did today. I would do wanna apologize for, some of the little hitch that we had, from a sound standpoints and from a Wi Fi connection standpoints. It happens. But it was not the, it it was not our target and ambition. We do hope this was, this was interesting for you. Don't hesitate to send us any questions that you have that you would like to ask. And as I was saying again, there will be a follow-up email that will be sent to all the participants if you ever want to go back into the material that was presented today. Thanks a lot for your time. Thanks a lot, Dhein. Thank you so much, everybody. Thank you, Sophie.