Video: From demo to defensible: Governing GxP AI at scale with Good Machine Learning Practice (GMLP) | Duration: 3184s | Summary: From demo to defensible: Governing GxP AI at scale with Good Machine Learning Practice (GMLP) | Chapters: Welcome & Introduction (10s), Dataiku in Healthcare (193.25s), Technical Setup Check (487.36s), Life Science AI Principles (538.69s), Platform Solutions (720.555s), GMLP Framework Overview (801.175s), GMAP Framework Implementation (1099.785s), Model Development Workflow (1249.985s), Production Deployment & Monitoring (1979.41s), Key Takeaways and Framework (2361.275s), Q&A and Closing (2860.735s), Closing & Next Events (3117.925s)
Transcript for "From demo to defensible: Governing GxP AI at scale with Good Machine Learning Practice (GMLP)": Hi, everyone. Welcome to GMLP readiness webinar. We're gonna give one and two minutes to, before starting the presentation. One one minute more. Time for more people to join, and we start. K. The room the room's still, people are still entering the room. Okay. So let's start. Welcome to, welcome to all of you to VJMLP, readiness webinar. As, as you expect, we'll be focusing in, in health care and life science today for the GMLP review and solution. Some logistic firsts. First, the webinar would be recorded and will be sent to you after the after the session. Second point, if you have some, some question, please use use the chat box, and we'll be answering your questions at the end at the end of the session. Be sure that we'll keep some time to be able to share with you to to to go back to you with you or with your point of of integration. And let's start by introduce ourselves. Well, maybe, Rene, I'd like to start. Yeah. Thank you, Miguel. Hi, everyone. I'm Remi Bach. I'm the AI governance lead for health care and life sciences at Dataiku. So very, very pleased to be here. And I'm, Michael Atlant. I'm heading the industry solutions team at the Lidia at, dot ai. Nice to meet you, all of you. So today, what would be, visiting around four chapters? We'll have a short introduction about, what that IQ is about and what that IQ is about in health care and life science. And after, we jump directly into the JMLP solution, first with our presentation on what is our JMLP solution and after a live demo at that training will be rolling out. And at the end of the day, some conclusion and keeping time again for the question and answers. So that echo is very well present in healthcare and light and life science sector. Today, we are selling most of the big pharma of the big pharma companies. You have some logos here on on on this slide. It's not it's not exhaustive. We are more than 50 than 50 customer there, and we are serving our customer around different dimension and functions. They are using that for exploratory or to scale up solution to a digital level. They're using that for data prep up to advanced agentic solutions, and they are using us from IT teams up to business teams encompassing finally, from r and d, early stage discovery up to patient and health care provider centricity. So a very large footprint on this area and quite, dedicated ex experience. Today, what happened with, with, with AI and probably some elements you're facing at your at your level. You might have more and more pilot scaling up on your on your on on the back burner. There is some challenge on, how to implement for strategic solutions, a longer regulated processes. See, life science, very some of our industry that highly regulated could be aerospace, could be banking, but life science is highly regulated, and time to time is often a blocking port, to be a GxP compliant, to be GMMP compliant. It's a point we'll be seeing today to answer, for example, IA act regulation. And, at the end, another point important life science is that we have in front of us a large panel of different solutions, data source, and and raising number raising number of agent. What will make a success moving forward in AI? So first point is to have pilot not sleeping on the back burner, but really pilot being able to reach production in weeks and being from d one implementable, including across GXP, GXP, processes. Second point, which is important, is not have a barrier between what are the domain expert and the data scientist. They should be able to discuss, to work together on the same platform to be sure that the solution that are built on where the needs of the the needs of the domain expert and that the data scientists have a clear understanding of of of of of of of expectations. Finally, the last point is to ensure that every AI output remain traceable. It's explainable. It's defendable because you will have to face, obviously, regulatory inspection as an example. So what's the specific of Dataiku? In fact, Dataiku will play the role of a bridge between what is your data stack and the and the and the domain. expert and the and the business functions. That echo will be connected and totally agnostic at the end of the day of the different piece elements of of your stack. So we can connect to different cloud providers, data platform, different legacy system that could be in place. You can be choosing the LLM provider you want as an example. So, if you build a solution that they could if your data are in Snowflake and if you want to use OpenAI, you will be able to put in place the right connector to build your solution on top of, on top of your of your selected provider. And we connect also directly to different enterprise app, like could be Vivo or Salesforce, as as an as an example. So in that echo, what will be possible, work in collaboration between domain expert enterprise infrastructure. In fact, the idea is to build altogether, was to build with AI assistant development tool. We'll be able effectively to orchestrate and connect to different workflow across across across the across the system and, across the different processes you might have in the in the company. After that, building your initial framework, you'll be able to deploy directly within data across across the organization. And the most important part, you will keep you will keep the control. So meaning, it's a solution which is used from coder people to low coder and no coder people to build the highest solutions, but we we have the being able to put this governing place that help to control everything what's happening to validate that each and any solution which is built within the data. We respect your guideline, and we respect regulation before being moved into into production. Yeah. Before we continue, Mikhail, I just wanna check. Okay. Okay. Great. Thank you, Melanie. I was about to ask because someone said that he could not see the presentation. So okay. If Oh, sorry. okay. From my understanding strange. yeah, Joe, just give give a ten seconds to see if people are seeing, the presentation. Okay. You can see it? Okay. Did you. saw the previous oh, okay. So this. Yeah. is, So that that was confirmed. okay. So maybe, Yeah. Mukesh, I hope you can see the the presentation now. Okay. Thank you, Arish, Okay. Perfect. for that. Great. Let's continue. Sorry. I did not read effectively the the messages there. Thanks, Ruby, for for for the warning there. Quickly, just for the recording, in case, in fact, you have it later. So if you do the slides, so my first comment effectively is the the companies, the different companies working with us, and I'm sure some of you are part of those companies. How moving from, today from pilot finally making AI success to different solutions. And finally, how the tech user interface within your stack and this overarching role, being your platform for AI success, being able to connect all the pieces of your of your stack with different with different, different function to build your data or your AI's your AI solutions. Why are we consuming life science AI difference? And, four elements that we reconsider from the very starting point in Dataiku when you build from the very initial pilot, you will be building Dataiku for your solution to, obviously, to the scale up of the solution. First of all, we want to have everything regulated by design. So, obviously, you all know that in life science, we have different regulation that are framing us, what we are doing in in data and AI, GXP, 21 CFR part two eleven, all what is around the DMEA AI prime principle, AI act included, which is not fair mentioned, but obviously, but for people based in Europe or having to deploy solution in in Europe. All what is on the stability, validation, change control, indeed, not negotiable. You need to have this in place, obviously, since since the beginning when you'll be developing a solution. The second point is about high fragmentation landscape we are facing. I think there is no other industry where you have so many different data source, so many, and, and structured document to work to work with. And we you you need to have a platform finally that will be agnostic of our different data source being able to connect to all of them. And if you change one of those data source being able to be future proof and, to what will is the evolution of your stack, not destroying the solution you will be you will have been built on top of it. The third point is about gigabyte effectively to have any output, which is evidence grounded. So to be able to, have a full regulatory, auditability and that you are able to trust back every approved data and, every outcome from the different, AI solutions you will be developing. And at the end of the day, to ensure that you will force to have a human in the loop. And thanks to the government model, and Remi will be discussing a bit more about that later, you will build effectively to ensure that when you are when you are in production, your solution will not move until the expected human needs to look validates the step you are you have you have you have defined an initial. I will finish with this one on my side. So what we are doing for customers, in fact, I was mentioning, it's a that Equisoo platform where from no coder to coder people, from the function to IT team, you will be able to build your data and agentic solutions on top on top of your existing stack. We were changing nothing there, connecting to the existing tech stack. And what we are doing also for our customers is to build solutions that are ready to be used, ready to be downloaded. So solution template, obviously, that we need to be connected to your existing ecosystem. And as you can see, we have developed a solution and this is a sample of it across the world value chain in life science. We're also supporting our customers in building solution with team of experts. And here in Orange, some example of solutions that in fact are GxPGMFP eligible by definition that we have been developing. So I will not enter into the data of this slide because it's important to take time to move within JMLP, but very happy to further discuss if you have a question about some specific solution you want to understand a bit more what we are what we are doing what we are doing there. Remi, the floor is. yours. Yeah. Thank you, Mikhail. So you can stop sharing your screen. I will share mine. So. that ahead. great. Okay. So you can see my screen. Right? Okay. So, let let's take a moment to to to to review what is, GMLP, meaning good machine learning practice and why it is something quite important, and I think we we found, the guidelines quite compelling. As first, it's, a framework published by the FDA, Canada, and UK MHI, publishing, joining issued principles, that were at first for only a certain sector med tech, and then there was applied to across all AI TxP activities. And what is quite interesting is that in the principle they're pushing, these are the same principle that we see with all the frameworks, but more related to licensing than the challenges that we are facing. First is making sure that people are the at the center of what we develop, that we're having the right experts in the loop, that we're having the right process in place, required good software engineering, risk management, and cyber security practices, that we can ensure that our data is trustable is trustworthy, meaning that, understand that our data is representative of the entity population, that we can separate our data into train and test sets to build our models, you know, in your case when it is machine learning and to make sure we have the right standards, and the the available on the available methods. And third one is really making sure that the model is fit for use, and it was designed with that. So, basically, that's we are testing the model, used with, the human AI team, that we understand how we're gonna, review the performance of the models that will lead to our fourth point, which is about transparency and monitoring. How do we make sure that when we're building, a use case for clinicians, that the clinicians get the clear information of how the model behaves, what are the performance and the limitations of that use case, and also making sure that when we push a use case in production, we can monitor it and have the right alerts in place. About, GMLP, Mikael, reminding what are the challenges. But, again, I think the main challenge we we see with, lots of our customers is that, that great, ideas, use case that we want to push into production in, GxP teams, in r and d manufacturing, but the right evidence is not captured, which blocks the innovation or you need to to build it, like, from from the the top down to understand what happened in your model, which takes lots of time. And for for a good reason, if, compliance and QA teams, they need to make sure that, the models are used, are built and used in the correct way before pushing into production. And they have to make sure also that, when it comes to data, we are very strong and we are very, focused on the integrity of the data from the source datasets to the model outputs. The solution we are, releasing is basically to answer these challenges. What it does, this solution, is really a life cycle solution that captures this rational data provenance, this evidence that are needed, by UQA teams. And when we take at the life cycle that we built in, g in Dataiku is you go from qualifying the use case, defining the the business view of what is the, the value captured versus a risk, understanding the scientific call, my model version with more technical information, and making sure that we have a deployed system, and then monitoring that deployed system in production. So if I go if I do a deep dive on the different sections, we see that at ideation, what we do is before the development, we qualify the use case. What we did with our solution is we are reusing the language of, the TMLP principles, such as the entities, the data feasibility, human AI interaction. And then once you move from ideation to, the development of the use case, you have the business case, which is, the, the overview of your use case. You can have one model. You can have one model and one agent. You can have two models and two agents depending on the use case. So it's, the bucket that encapsulates these different, models. Then you have the model version that is, the big piece, which is how do we, understand the data strategy, what were our development tests, did someone validate, that model, how do we what's the deployment strategy and and then the monitoring part? And the AI system is really in terms of, okay, what is the environment we're targeting? What is our post market monitoring plan? And what are our UAT and release sign off. A good example for GMAP, I think, is Judicial Twin. That's a UK use case we we built. So we applied GMAP to that use case. And I I think it's a great use case because, digital twin is, a use case where, that if you do something wrong and if your model is not working properly, you don't have the right documentation, you would have spent millions of, dollars of euros in having the digital tool twin, making the the investments to, maybe to try producing that drug for at at the end of the day, not being validated by your health authorities such as FDA, EMA. And so in that use case, what's interesting is that you have a a model of machine learning and an agent. And our GMAP framework is we have one framework that is, adaptable based on the technology. And so what we'll see now is the framework that we put in place from ideation through monitoring, in our platform govern. So the end to end process, basically. So here I am in my, digital twin, my ideation stage, meaning that, before developing anything, I need to fill out some key information. And so here I see that the overview is basically information, key information for my workflow. When I click on the workflow itself, I go to the business strategy, and I see that I have, some source documents. We know that companies, you'll you'll have, you'll made documents of, an ideation of that use case or protocol that might be deleted. And instead of filling out a work workflow manually, what you can do is adding this PDF file that you have and calling in a given agent that's gonna draft from these, documents. So, basically, when you will find the information, it will, fill out, the field that you can see here by such as clinical, DXP, the geography, and we'll see moving forward the different categories. But this is the first option of filling out the workflow, calling an agent based on your files. Second option is retrieving from external system. We know that some informations might be on, other ticketing tools like Jira or ServiceNow. The idea is really to call the these systems or collect that information to put them in the right fields. And so if we move forward with the business strategy, we see that we, the agent defined the value proposition. So my role is to review it. But, yes, I agree the problem statement is exactly that. We're trying to, to to build a control arm, so based on this, digital twins. These are the defined desired benefits, target population. Here we defined what's the operational definition such as the entity choose, how it's gonna be integrated in the workflow, then how it's gonna be deployed. For example, for a case of, manufacturing, maybe it's get deployed in one factory or in different factories in, different locations. Then I can check if I have the right experts. They're gonna be available for, my use case such as the data science lead, the regulatory, and the domain expert. I know that for this use case, I'm gonna use agent in AIML. If I have, available the ground truth data, meaning the dataset that we're gonna we're gonna use to to start the project. I can add more information about that particular datasets such as, well, from which system, all my source dataset is gonna come from, what's gonna be the data store description, the how we're gonna assure the representation in that data, and we have critical and subgroup, that we need to identify. And then, what's your independent strategy, and what's gonna be all integrity controls? Then we move forward with the risk and human AI, meaning, how do we foresee the human AI synergy? This is the vocabulary of TMLP, mentioning human AI. Quite interesting is, okay. We're gonna build the use case, but, is this gonna be a chat interface? What's the interpretability needs? Something that we will, reuse, during the development, to the monitoring parts. Then you have more information about the technical safety and security, some risk identifiers, which automatically create your risk registry that would follow, throughout your life cycle. With a rescoring based on our calculation and a risk level. If I'm not happy with that risk level, again, of course, override that decision. In that case, that's what we did. You know, we didn't take into account that information, so, we put that as critical. Once we're good, we sign off, and then you select your ML platform you want to develop. In our case, we selected DataQ. You could have selected any other platform you you develop. You click on create project, and that will create, the project automatically. So before I see the governance on the project, I will see the the digital twin, projects. So, basically, this is what happens in the Emerald platform in Dataiku DSS. What you have is you have the raw data. You have the, data preparation where you clean it. You train your model. You test on the different subgroups you identified in ideation. You do the the matching of your digital twin, and then you build your agent, that, your clinician is gonna use with, some evaluation store, which is making sure that we have the right metrics for that and other dataset that are very useful in in case of audits. In parallel, the governance exercise, is been done, meaning that while building the evidence, are captured over over time of, the development of the use case. And here we can see that at the risk tiering, we have the effective risk level. The residual risk, the effective risk is something that won't change over time. Meaning, the defendant ideation, this won't change. However, the residual risk will, change depending of if the validation, has been, given the green light, if there was an issue in monitoring or any software engineering control that, was not done. This call would go high or low depending of, what happened in your MLOps platform. And, also, it could send an alert to the right person to to let know that this use case became a high risk in terms of residual risk, and here are the control that we need to do. So here, we see that this is, related to the, business initially, meaning the unit of clinical development. We have our GMP ideation that is linked here, and here we have our model agent. So let's get through that. Going through the agent, I see that in the overview, same, I would have, some key information, to explain my agent. When I go to the agent scope, I see that, I have an AI autofill, meaning that I have some questions to fill out such as the agent purpose, the operational safety control, the data provenance. So instead of doing that manually, what we do is we call an agent that will check the workflow, check the information that it needs, based on the different fields, and it will fill out the information for you. So, basically, when I go to data strategy, I see that I have my source datasets. So what it did, it went here and say, okay. These are my source datasets. I checked if there was a PII tag. It checked also what the, agent, or is producing in terms of, output datasets. And then you can see that they are information that are collected from ideation, meaning we retrieve automatic information from the ideation, what we define the last stage. And then the agent is saying, yes. If it is aligned with what's, really in your, in your project versus, what was defined in, ideation. Like data source description, yes, it is aligned. If it was not aligned, there will be a field that would request a user to justify why it is not aligned. Same as data quality, something we we mentioned, data integrity is, is key to, GxP DMLP. So the agent is doing that based on the different datasets. It's checking also the data quality checks, we have in the project, and it also collects the, the more graphical view of how my agent is connected that we could see, in a beer. When we're good with that, we validate the data strategy, and we can move to the model development. When you can see that we have the element card that is, that is used, We also have a framework of that LMM card with more information of it. Probably, we have different models using GPD, 5.4. We also have the prong change history with more information about how this prompt was deployed and if it is still active. I see that I see that the v three is the one we use. More information about the guardrails in place, the different tools that the agent is using. With same a bit more information. If also, here, this is a multi agent or structured agent as we call it at data queue. So, basically, what did is said, okay. This is these are the different steps it take, and here is the flow that, we could collect from the agent. So, basically, very aligned. If I go back to that image, very aligned to the different toolings, we saw in the table just before, but, we see the tooling here, and we see the end to end process of that agent. Then we have all the information such as the memory, the methodology of the agents, and the model development metrics. And once once it's good, we have a field that is filled out as, okay. Everything is passed. You can move and sign up the model development, or you could, sign off the model development automatically if, if it is, the use case. In that in that case, it is a critical use case, so that's why you see sign off that are manual. And, also, we need to have a sign off before going to production as from 21 CFR part 11, but still sign offs could be automatically defined. Then you have the independent validation, which is, again, you're calling an agent that's saying, okay. We did this test on the agent eval prompt. These are the metrics that we collected, we passed. So, we move forward with the sign off, and we could go to our deployment. So the bundle is an AI system. So, basically, you see that use case, I'm not gonna have only one agent that I put in production. This agent is dependent of that model. So what we will do is we'll, bundle all these models and datasets and recipes all to work in production. And for that, we need to make sure that we have the right target environment. Again, what we do here is that, we call an agent that will check the bundle, meaning a bundle is an image of your project, so it's immutable. You can modify it. So what the agent does is is gonna check this, this bundle and gonna collect the the information we need, that is required with GMLP, such as what's the target environment, the cybersecurity controls and data integrity controls. In that case, I see that this is a match compared to what we defined in ideation. Same as here for human AI readiness, collecting for what we define in ideation, checking with what we have. We see here this is partially aligned, and, you have the reason of why it is partially aligned, or you could modify and add more information to justify that gap. Post market monitoring plan is a big one, as we need to define what are gonna be our monitoring of the, the system before we push it to production. Because we know with models and agent, the data can drift. This is usually what happened the most. You might have the performance that is not accurate. So what are we gonna, target? And what are gonna be the alerts, that we define based on when there is an issue, who should be contacted, what's gonna be the, the, either gonna be a restrict or art mode execution. This information I keep for then later to be able to monitor correctly your use case. Then you do your UAT. So you have a release manifest summarizing what you have in the bundle. So quite, a bit of technical information, but, like, what's, is your flow composition, what are the transformation chain, and the data quality checks. Once we go with that, we also have, what we call project standards, which are checks of the projects, meaning every project that are of that type, they need to grow through, certain, standards. And here we see that we have too many datasets, but that's not a blocking, standards. And so before signing off, also something that is required by JMLP is called instruction for use, meaning, the clinicians should have a document that, how should do who should they use the, interface, what was the target population trained on, how it's integrated in the workflow, what was the data used in the node limitation. So basically to give them the keys to make sure they can use it in production, in the right way, in the most efficient way. And so here, what you can do is you can extract, this I IFU in PDF so you can share it with your clinicians just after, the go live. Here, the risk profile, you take on the different control that can be checked for you basically is what we are we defined automatically in ideation. So we make sure that, none of the risk we identified are forgotten, before we go to production. And here are the different assets. I I see that the status is good. My agent and my model are good, so I can sign off. As soon as I could sign off, I could go to production and, having my model, running. For the monitoring parts, so when the once the model is in production, you have some status about the data, the data drift, the data quality checks, and you also have information about the, the metric values of how they've evolving. In that case, I know that I have a warning. I can also have, an understanding of my alerts, if they were triggered or not. I see that I have a warning also that I could go deep dive to see more of it. And, also, what I can use is my, production models console. So, basically, I want to understand the health of my agent, in that case as with more information that I can have in the workflow. So I can see that actually have a warning, but the data drift is normal. So, basically, yes, it is a warning, but there's not not any action, especially that warning was about the average tokens, that is a bit high. And the last stage that is, also, forgotten in lots of frameworks is the revalidation part. We know that at some point, your model, will be drifting. So how do you make sure that you can revalidate it? And not even we know that, at some point, it will be drifting, but we know that even if the model is not drifting, a good practice is for a critical, a high risk use case is to review it, in a in a time framing. We see some customers reviewing every year, every two years, depending on the use case. And so, basically, what we do here, we revalidate some information such as, the data. So here, the ground truth. We check also the different metrics for that revalidation pass parts. I see that is passed with conditions and that, I can mention a justification. Once I'm good, I could save that revalidation, meaning it's gonna be, it's gonna clean that step. And we're gonna have the revalidation, meaning that the more you revalidate, the more you're gonna have rows in that part with the right information of, how it was triggered, heat, time triggered, by who it was signed off, what time, and more information about that particular stage when it was revalidated. Also, something interesting is, this workflow, these workflows are fully, extractable, exportable in PDF as what we are doing here. So, basically, what we did is we collected all the information in the workflow, from the ideation to, through the project, the agent, and model. What you could do also is if you want to send that PDF or part of that PDF to your QMS, this will be feasible with a connection directly to, the, platform you're using. And so that was it for the, TMLP demo. Coming back to the key points we like to highlight, for the key takeaways that we have for, GMLP. We really think that this framework is quite powerful because it highlights a discipline rather than, only principles. And the fact is we have a new framework from, the FDA EMA that was published recently, about drug development. And when mapping the framework of GMLP with that drug development, that's a more recent framework, actually, it matches really well because what we want to understand is to bring the governance from the data to the model, to the agent, having the right, documentation proof that is automatic, that you have human in the loop when it is needed, just to bring that innovation, making sure that you have the right evidence that you can scale faster. Because it's very different from, having three, five use cases that are GxP that you can rebuild the evidences, compared to, twenty, fifty, a hundred, two hundred use cases. It's where we're going even more. So having the right process in place that you can trust is really the key to, escape faster. So a couple of, points that I like to highlight is really tie out ideation, meaning define the risk hearing, before you develop. So you make sure that you have the right, the right understanding of that use case, to put the right controls and asking the right evidence, meaning a short short use case, a use case that is, not impacting in a certain way, that can be just of, reviewing some, commercial that not not even GxP. It should not go through that, more complete process. But something such as digital twin, which is quite impacting if you make an error, that should go through, the complete process that you need and that you consider, enough for you. Second is name an accountable owner, meaning make sure that from the from the ideation that you have someone who is gonna be the the responsible one to making sure that we have all this information captured and that we're gonna be able also to handle the monitoring parts. And the third one and the no not the least, I think I've I've highlighted that quite a lot in the webinar is plan for drifts. It's what we see the most, that drift would happen. But when you put that in an expected event, it's when you you like you have the keys to make sure that a drift is something that you can handle and just retrain the model and, putting that back to production to bring in the value, you want this mod this use case to add. Okay. So we're arriving at the the last slide. Mikael, do you want to add anything, until now? Not not that much. So, well, I hope first that it was quite, instructive for you that you, that you saw the the capability and the and the interest of the solution, particularly again for life science, and then it could be applied to other to other sectors. We stay available anytime for further questions. If you want to look deeper onto it, if you want to have if you are not yet using Dataiku, you want to better understand how Dataiku can help you being on the IT side or being on the business side, obviously, can help you to expand data and AI and advanced strategic solution across your across your organization. And also, finally, we would be very happy also to come to you and to discuss about governance of your AI solution and how to ensure that you are compliant and you are able to deploy voice solution on GxP processes. I see that. there is no that much question and answer of the any any question? We get one. Curious about change management works when traversing between traditional systems like legacy QMS, the system. Remi, I'd like you to take this one. Yeah. Can can you read it out loud? Have, Yeah. my. my speech shared? the question is, so the question from Melanie, I'm curious about how change management works when traversing between traditional systems like legacy QMS, quality monitoring system and this system. That's a good one. So, basically, what we bring to the table is a framework to govern from ideation to monitoring. And so what we have to understand that we what we do with customers is understanding what kind of information you need in these legacy systems. Basically, when you talk about the QMS, how do you what kind of information do we want to have in the QMS, from that governance layer, from that governance, exercise, to make sure that it's not only a system that lives on the side, but it's something that is integrated in, your ecosystem. I think it's what's the most important is really seeing that as something that's what we see with customers is, integrating that as part of your broader scope and not something that, should just add manual steps to, your process. Like, that should be in your SOP. That should be something that is, known in your organization. So and I think that's also what I see from, from a experience is that when you want to this apply to, we saw a use case of data queue that sign, but we could apply that to any use case, not only Dataiku use cases. If you have any, other use cases like such as Azure, model agents, we could integrate this. And what I see, probably when it comes to legacy systems such as SaaS is making sure that you can, talk have a talk to, govern to that system. Because sometimes what we're facing is that, in some version of SaaS, it's a very closed system. So it's hard to collect that information, and we should have a way to, to basically have that information flow, in the both ways. I don't know if I answered your question, Melanie. And if not, feel free to to, to to just, send another message. Since since it was fine, we have another one, from Julia. What's the smallest platform change that actually moves the needle at thirty plus years? In this case, Julia's point is GMLP or globally as a platform? Okay. Let's move for JMLP first, Remi, and after we can JMLP. Okay. Thank you. So Yep. So the question is, what is the the first step to do? This is what's. No. What's the smallest platform change, Vikki, that really that actually moves the needles, when you reach the the those, 40 plus use? Think it's the smallest one I think the smallest smallest one, will be to have a sign off before going to production. The smallest one, saying that, okay. If you want to push something in production, you should have a review for high and critical use cases. So if I take the end to end process of what is the piece that is, really, like, the first step of, okay, what's because it's requested by regulations also 21 CFR part 11. But, another small step will be okay. What's our minimal process we can put in place to start? Because what we are showing, we are showing a complete solution, that is very aligned with the principles, taking into account all the PC that's, required and will be, requested by US authorities. But I think there's also a first step that is, okay, how do we define a first draft or smallest, process that we can get used to, just having that process, filled out and and because it's and so also so another way of thinking of how do we document, the from the ideation to the monitoring. So not going into, taking the plane the the plane first, but having the bicycle of, okay, how do we how do we manage to integrate the smallest process in that? And then, bringing more complexity to it that is required by regulations, that is required to understand the efficiency, but always starting small. And maybe if I may ask, add one, so there is different smallest changes, but one could be also to really think about GMLP about compliance is a very, very beginning of your project. So don't think about it later on. Start to encompass it from your POC, post consideration that across the platform, you will be able to effectively to build Cesar initial POC with minimal burden for the people developing the solution. So you will not be catch by surprise after that when you want effectively to expand the use, finding in front of you roadblocks that were not contemplated at the at the original stage. So something which is very small at the beginning could become a definitive barrier at the time you want to you want to industrialize. Yeah. And I think for the process to be, sustainable, it needs to bring, some automation to the process. That's why we've been showing, through GMLP. We have lots of automation capabilities, agent capabilities always revealed, needed to review by, a human, of course, especially when the case is high risk. But I think, the evidence should be captured as you go and not entered manually. They should be entered manually when you don't have anywhere where this information is, or could be collected or it's something that is really, like, from your brain thoughts that you need to put on paper, then yes. But so that's what we we do with customers is really making sure that, this process is integrated and bring value because that's what we want. We wanna make sure that we comply the regulation, write the right evidence, and that we make, that innovation, even faster and safer. Any other questions, comments? Seems no. I'm just looking at the messaging system. No seems no. Well, thank you to all of you for for for screens a dedicated to succeed event, so you can flash the QR code that will drive you directly or you can find it on our site that will take place this month, September 24. Romy and myself will be there indeed. And also we'll be managing the session dedicated to compliance in life science and how to scale up your, your agentic solution in life science. So we'll be very happy to see you there in case it's possible for you to attend. If you're based in France, there is a week after, there is that I could submit in Paris also. You can find all the information in in in our site, where also different topics will be would be would be addressed. Yeah. Exactly. Thanks to all of you, and, Yeah. Thanks. don't hesitate to reach us back for any further question. And, again, so we'll be receiving, the recording and the slide deck. Thank you. Goodbye. Thank you, everyone.