Name: Hyper-Personalization with AI & LLM Agents
Uploaded: 2025-04-10T15:32:50.990Z
Duration: 43 min 8 s
Description: Hyper-Personalization with AI & LLM Agents

Transcript for "Hyper-Personalization with AI & LLM Agents": Hi, Yiran. We will be starting shortly. We see people joining in. Thanks for being here today. So today's session, is about IPI personalization with AI and LLM agents. And I'm happy to cohost, this session together with, with Quentin. So we start with a quick intro on my side. We'll let Quentin, introduce himself as well. My name is Bernard Broja. I'm AI solutions director for retail and CPG at Dataiku. I'm based in Paris. And we are really happy to organize this session about, LLM and AI agents together with Quentin. Who can now introduce himself. Hello, everyone. I'm, Quentin Goyens. I'm a solution data scientist in, for the tech who in the session and, for retail and CPG. And I'm working on developing the projects that we'll see today and among a lot of other projects that we have available for orchestras. Thanks, Quentin. So first, starting with a very quick introduction of, Dataiku unique value proposition. So for the ones that, do not know us yet, as you get to know, what we do at Dataiku. So at Dataiku, we're a universal, AI AI platform, that is, tailored for, everyday AI. We both handle no code and full code capacities, exploration to production environments, and, analytics and data science workspaces. That's very very much the core of, Dataiku's, value proposition. And we also made, generic capacities, of course, obviously, even the topic of the day. We are a GDI ready platform, with, the LADAiku LLM mesh, which is, as we will introduce a little bit further in on the next slide, is a common backbone to GDI services, with, multiple prebuilt and, reusable components to accelerate, you know, LLM, developments, and then projects, developments, and also some performance cost and security control layers, in order to have all that, governed the right way. And speaking of which, we also we also have a govern, governance layer, as we act as a central control tower of all data, machine learning, DNA products, in full compliance with, governance and regulation, with capabilities to, monitor, approve model deployments, as well as qualify project value and and risks. So that's the, you know, the core, value proposition of, Dataiku. And as I mentioned, I'm gonna quickly deep dive onto the very, centerpiece, which is, the the focus of the of the topic of the day with, Generated AI and, agents. And so we offer a marketing generic delivery platform. So what's part of this? First, with what we call the LLM mesh. So the LLM mesh is a set of connection services, secure gateway, approach to allow for, connecting to your preferred, LLM API services or self hosted, LLM, or Victor stores, that you are using with the modularity that helps any company to avoid, locking in a very evolutive markets. On top of it, we have what we call LAM GUARD GUARD services, which is some kind of a first, governance layer. But as you can see, we also have a transversal governance layer as I was introducing in the previous slide. And with guard services, we offer the ability, for enterprise to monitor their LLM, costs, also to implement, safety guardrails as well as quality guards, on the ways they use, LLM in their different use cases. Of course, all that, is the backbone of, builder, accelerators in the form of, you know, building use cases. And so we do have specific accelerating components, as you can see on the slide, with LLM visual recipes. We also have code recipes, of course. We have, test environments such as what we call prompt studio where, anyone can, you know, test the, ability of of of a model with prompts, to validate the efficiency to also estimate the costs of specific requests. We provide with some, advanced, interfaces, which are called Dataiku answers with the ability also to go beyond just, you know, simple chat and and with, the ability to connect, several agents. So it will also part of the demo of the day. We have the ability to, build, LLM agents, obviously, with code and the code and no code, as well as, fine tuning visually LLMs. All that, as I was telling you, being governed transversally with a governance layer, to keep pace with, you know, evil evolving regulatory requirements, should such as, EU AI act, or LLM and Gen AI registry capacities. So that's, very much, the breadth of, Dataiku offering when it comes to, Gen AI, overall. So now going into the topic. Today, we'd like to as as we initiated, with the title of the presentation and of this webinar, we'd like to go through a quick intro of, about LLMs and agents in the context of written NCPG. So we'll start with a bit of theory. I will be, I won't be too long on that one. Don't be worry. But that's and then, we will transition smoothly towards a practical demo, with Quentin running it, right after, my introduction slides. So first, let's recall for everyone in the call, what's our definition of AI agents. So we see many different definition on the market. So we try to be, you know, for this session as pragmatic and simple as possible. So first, it's the LLM powered system. It's designed to achieve objectives across multiple steps as opposed to, generation bot chatbots as we'll see, on the on the previous on the next slide. Sorry. And third, it comes with a concept of going just beyond answering, but more like acting, executing some tasks, with autonomy being either full autonomy or partial autonomy. So that's what we consider being, an AI agent, you know, with those three components. So now what, difference do we make between Gen AI chatbot and AI agents? As I was, quickly touching on, in the previous slide, generic chatbot are more about answering questions, and providing information based on a knowledge base, what's usually called RAG. And for example, we would have an expense policy chatbot that will help answering internal questions, like, what's the limit for client engineers, or do I need receipt for taxi rides? Simple questions, detailed answers. Detailed and accurate answers. Agents, goes one step further with the ability to plan and execute multiple steps, as I was telling you, across business systems and tools. And the latter are very important across business systems and tools so that they will act they will perform actions on connected systems. And so, for example, an expense report agents will be able to review each each each receipt, We'd be able to check, those receipt against company policies with autonomy. We'd be able to flag, any problem attached to a receipt, to route items to the right approvers, update finance systems, or even, like, open a thread with employees and, you know, to to following up if needed. So we can see the diversity of tasks, that the agent is provided with, and its capacity to interact with external, services or or tools. So now transitioning towards, once we have set the scene, with, what's, our agents, for instance, towards, what we see on the market, when it comes to agentic use cases in retail. As of now, we have structured the presentation across four main domains. It doesn't mean those are exhaustive. Those are the main four domains we are seeing being discussed with our customers on the market when it comes to implementing agentic use cases. The very first one is what we call augmented augmented store manager. It's basically going from sales support to stock management. The idea being to, you know, support, customer interaction and management of related supply processes, for example. And so in this context, an agent would be, you know, supporting a sales assistant or store manager by offering the best options and information related to multiple scenarios. It can be about stock inquiries, it can be about cross sell, opportunities, it can be about any any information, that can be, that can support a store manager in its, operations. Second main area we are seeing, being impacted with, agent EQS cases is IPI personalization. Here, of course, we are talking about AI driven marketing with AI agents having the ability to deliver personalized promo, analyzing browsing and purchase data, for example. And one, you know, way to, have agent act in this context would be an agent to identify key customer characteristics that would dynamically adjust recommendations, and offers accordingly. Third area is that it's is, is tied to counterfeit and fraud detection. And so helping companies better identifying counterfeits and fraud, which usually is a painful process, of course, involves, you know, big, big, you know, economical stakes. And so those processes can be augmented with agents by, for example, scanning online assets, such as marketplaces, social media. So I was talking about tools that would be specific tools provided to the agents so that they are able to go after those scanning activities, around market marketplaces and social media, for example, to spot fraudulent activities linked to any unusual patterns. And so in this context, agents would be analyzing transactions, to identify anomalies, and generate, any, you know, related, reports with associated risks and recommendations. And finally, the fourth domain we're seeing, impacted relates back to rewards and incentives. And so we are talking more about optimizing loyalty programs in this context. And so goes, also back to, how much companies can personalize loyalty rewards and can engage towards personalized strategies. In this context, an agent, would automatically processes, process or customer customer reward requests, and we'd be able to interact with external external systems, such as CRM systems, well, company system, of course, so internal, in this context, to update their profiles, with new information, new contact information, or, you know, to interact with these, different systems more specific. Very important point to have in mind, I mentioned it earlier, but, better reminding it once more here, is that for all those scenarios, agents are equipped with tools. And tools, again, are about the ability to perform, for example, on an online search, to write SQL queries, to access customer segments. They're also equipped with datasets, which can be, you know, simple data tables that are connected, to to to those agents, and provided to them, which can be also in the form of CRM systems. We can be related to back office transactions, you name it. And finally, they are also, you know, able to, provide answers, of course, but more importantly to, you know, execute those tasks and provide answers autonomously. So that's the important piece of this, latest part is the autonomy, which they are, able to bring. And if we try to, let's say, split them into some main categories, what we see is that, we find three main categories in terms of, use cases. We are first, what we call adjunctive workflows. Adjunctive workflows are really much about automating processes. And so we have just, you know, segmented the previous example into, into use. And so here, in the way we have, you know, named, and and told a story about these examples, we would have counterfeit and fraud detection as part of process automation because agent in this context will help to automate this painful process, to spot fraudulent activities autonomously, and very much ties back to this, automation of processes. Same as for handling rewards and incentives, would fall into this process automation flow. The middle part is about productivity agents. So productivity agents help companies augment workers. In in the example error, which we shared is around IP personalization, where we would bring agent would bring the ability to enrich traditional recommendation engines with LLM generated content, and that will be, part of the demo content we make right after with with full control. And with full control is also a very, big big part of the objectivity here. Last piece is what we call decision agent. And for decision agent, we lead back to what we call enterprise intelligence. And the example attached there is augmented store manager, which provides, so the agent provides, resolution cases, with optionality to sales managers, and can perform related actions in logistic systems. So we see back and forth capacities, between humans and systems, with autonomy, from the agents, being unlocked in this example. So those are the three main areas, which, we can, you know, which we can distinguish when it comes to, agentic, agentic, use cases. On in the bottom of the of those three areas, we have just, illustrated what's, KPIs or, let's say, target can be, attached to, each of those, whether it be cost based reduction, whether it be bringing more effectiveness, or or or or just, you know, increasing revenues. So now let's, do not wait further, until we go into a practical demo and we see an organic workflow in action. And prior to, going into the into that I could, as Quentin will share his screen, I will let Quentin just introduce the topic, with the next slide. Oh, sorry. Yeah. So, yeah, the topics that we have in hand, today that we want to share to you is basically the idea of what could be, powerful agent associated to, marketing, workflow. So as Benoit said, the idea is to have a no mounted worker system in this, example. So the idea is to have always a human in the loop and make it easier for marketing analysts to be able to communicate with the different datasets as a different, interfaces that you would in the regular life. So instead of having to know all the datasets themselves, what are the columns, and what are the content of every information of, like, data transaction data, past history of customer, they would just have, a generic communication, in natural language with the agent that would then, dispatch those queries to the different, subparts of, architecture that we have in in mind. So here we see a simple example where we, marketing analyst wants to send custom email to, likely to churn customers. And so the agent will be able to understand what the query means and say, okay, I need to identify what are the customers about, possibly the lowest future customer lifetime value, Identify them, match them with, valid product based on the product product recommendation, datasets. Think of characteristics from the product and the customer. Write a sample email. Make it review to the marketing analyst to make sure that everything is, in order. And then if, the email is valid, maybe, we go back and forth to modify some things. And then the agent has access to tools to send directly the emails to the customers who are looking up, customer emails, for example, and then sending through, emailing tool. And all of that as we we'll be able to see in the demo just after will be done in a simple chat interface. So it's really basic example, but it really shows that you can have, really, leverage all the data that you have in your entire use organization without having to really put yourself directly into each data set, which can be a challenge for business users. I don't know what we have in the next, slide. There's there's exactly a demo. Yeah. Kota, you can go for the demo. Okay. Let me make a quick there we go. So here, the the project that's really shared to you is really based on one of our business solutions that is called product recommendation. And it's, basically, an example how you can start from a traditional machine learning, data pipeline that could be, established process that you have here in that I do, and then how to plug it with LLM and agents as we saw in the previous slides. Here we see the basic, output from the product combination solution. So we see the past purchases for our customer, and we see what would be the, automatic project, that the model, want to to propose to a given customer. And the idea is, at the end, to get to the email that I mentioned previously. We don't see it properly, but, basically, it will be a custom email that, would be built based on the customer characteristics and the product characteristics. We will see an a better example later, so it's not too too bad. But it will be will is the topic of API personalization, or do you make content that is adapted to every customer, every product interaction? And so, obviously, you cannot do that, at human level, and you need to leverage LLMs to reach that kind of results. So to do that, I'll do a quick, explanation of what we do in terms of process from a direct perspective. I won't go into too much of the details. But the idea is to start from transaction history, which will basically be past processes of the various customer. We'll do a set of transformations. So every, blue square is a dataset and every, round, bubble is what we call a recipe that will be easy transformation that you can apply to your datasets. And, the green ones are what we call, visual recipes. And so they can let you do transformation very basically in the interface itself, so you don't have to actually write any code. So it's accessible for everyone. And we will assign this type of operation to basically do a preprocessing of those features, try to eliminate products that are not, that are very rare so that we have the most optimal, training later on. We do a step with collaborative filtering using some plugins to define the proximity between different products based on the interaction with the customers. And then we aggregate all of that into one feature dataset that we will use to train a machine learning model that we see here with the green diamond. And it's a good example that here we really have a full control of the operation compared to, other tools. You can look at every dataset in the middle, really check what is the data. So you if you have question of data quality, data control, you you show that you can go into all those details. Same for the machine learning models, it will be on your instance of that echo, and so you can exactly control what is learned by the model. You can learn what are the features that are more important for the recommendation system and evaluate the performance of the model, with any kind of validation data that you want. So at the end of this, green part, we get the results that we saw in the web app before. So, basically, what is a set of products with a recommendation score, and the highest score is the more likely the recommendation. But that's not really something that you can act on directly. So the idea is to go further and to be able to do an act to be able to activate customer based on this, information. So that's what we get in this, LLM part where we'll basically be joining the recommendation product based, on the model and with every kind of characteristics. So here we see, some of the characteristics that are based directly from, products themselves. So we have information of product categories, product type, colors, and so on. But we also have, external information linked to the customers. Here we see in the black square, it's datasets that are coming from other projects that have been imported. Because that's important to see that even if we see here one flow, and that I could have potentially a lot of different project running in parallel. And so we can share projects and data from each other models and use outputs from one, from one solution, from one project to, to be inputs for another. So you can really have, a global approach and tackle a lot of different, subjects in parallel. So here we have some segmentation data sets. So customer lifetime value and RFM segmentation. And with all these characteristics and customer segments, we'll be starting to write what we call a template, an email template. That basically be the, general, email that each customer will receive based on their segment. And so the idea is to have a limited number of templates that will be validated each time by a customer. Here we see, the validation, the editable datasets. So you can manually go into, the data and modify if you see, any kind of things that you want to modify. May we modify the wording, all things are presented, etcetera. And so this content has been generated by the LLM for by your first step where it's basically using, the customer, segments and some characteristics to build a template. And you'll see that we can, specify any kind of information that we want to make sure that the prompt is, is valid and giving you a good answer. In the same way, we will be generating a custom content. So limited, parts that will be very customized based on the characteristic of the customer and the characteristic of the products. That's really what will be the personalized aspects of the template. So because we do it on a small, set of, text will be shows that we have good creating that content and that we can use it, at a relatively large scale if needed. And so we can check every step as a content to look at a specific, result. It's one thing that is available in, in Dataiku is what we call comp studio, which is a way to iterate on point engineering in the most efficient way because you don't necessarily want to run, every queries on large datasets. It can be costly. It can take some time. It's not necessarily very efficient. And so you can do it in a prompt studio where we can easily add, features from the dataset, modify the prompts, and see what would be the results at the end and then we run it on a very on a small sample. You can also have an estimation of cost, which is very important if you want to have process as you'll be running on large data or very, often based on, your use case. And then from the prompt studio, you can save and export them back into the recipes that you're using in the flow. So you can really have iterative process to validate the content and then put that back into production system in the flow. Last step, on the LLM aspects is that we have, LLM as a judge system where because we are generating a lot of different text, we want to be able to validate that the content is proper, and then we do that through the LLM as a judge that we'd be evaluating the result from the previous LLM. As that way, we can emanate a lot of the different, content that will not be valid based on whatever works we want. This can be also be a stage where we want to review wording, be sure that you are using the right matter of speech, elemental language that you want to match your brand and, the want to the way you want to communicate to customers. Lastly, we'll be simply merging, what we saw the template and the content, the custom content, and and which the final product set will be as a email that can be sent as well. So the pattern that I showed that is basically the intermediate system where we go from fully, data content of the public accommodation to an actionable results that is email to send as a dataset. But the pattern that I showed earlier would be another pattern when we want to work at a smaller scale when you don't really want to reach thousand of customer at a time, but we want to do something more custom and more manual. And so that's work would be done through agents. And so here we have an example of what would be, an agent interface. So it's a chat interface where we have, we're able to ask question, and you have the history of conversation on the left for the previous topics that we we talked about. And we we here we can see, the type of questions that you can have, with an agent. So here we can ask what are the best selling products, search for the common catalyst of those products. And we see that we don't have to be specific, about the type of questions that you want to ask. We don't have to know where the data is, what type of column should I filter on, how do I define the characteristics. The agent will be able to look it up into the datasets for different, tools, for example, doing their own SQL queries, look for, description of the columns and so on, data types, and perform itself the operations that it needs. Another example here, just simple data lookup, what customer has the highest customer lifetime value, and we'll be able to do that very quickly, for the interface. So you don't have to look into the datasets and filter the information that you want. Another example of the process that we share is like a fully automated process. If you don't want to have the email in a loop, you can just say yeah, from the location, here is send an email for the best product for user, whatever code it is, using the customer metadata, and you will receive it, through the emailing tool. So here we can see an example of, products that will be sent. And so you can build through that a cool a set of, tools both with code and visual tools that will allow you to build, whatever type of agent you want as custom or generic as you want to ensure to match your your needs and your team's needs and to have, to have it connected to whatever type of interaction you want. Here, we just have data lookup and sending emails, but you can imagine looking into logistic datasets, making connection to, purchase of the, marketing, monitoring tools, any kind of operation that you want. Here you have the last process that is more like a back and forth, Example of this case where we ask what are the characteristics of the best selling products, get some results, and then we'll see, okay, what are the characteristics associated with them. We look for customers that could be recommended, and then we ask the, agent to draft an email that we can then review and ask to send. Yeah. I think that's the email that's sent, we saw previously of just removing one line that has the amount and, getting the results, you know, when. Benoit mentioned, monitoring and control, and one of the many ways to do that is, Trace Explorer. I won't go into too much detail because it's a bit more technical, but, because agent can be a relatively complex architecture, and we saw in the previous slides, multiple agents with multiple tools doing a lot of different things, it's important to be able to monitor what is happening. So here we see for all the past week as we have made in the interface, we can see the different, operations that have been processed, and we can review every intermediate, steps. We can also monitor the number of token, cost associated with that. You can also, log the time of every operation to be sure that there's no process that is taking more time than it should. Does that allow your technical team to be sure that the tools that are available for your marketing team is managed in a proper way and, matching the needs that you want to have. I think, that's all I have for the demo. I will I think we will go to the question. Unless Benoit, you had there's nothing else to mention on the project itself. I'm fine with the the mock. Thank you, Quinta. So let me just show you. Yeah. There is a question about the the trace, Quentin. Is it possible to trace the execution of the flow and which tool is executed? Sorry. In which tool is is executed at which time stamp based on which input and on which, and which output is generated by the tool? Sorry. I mean, what yes. I mean, what we call the flow is what we see here currently. So the different, the data management, like the data process, itself. And for that, you do have monitoring for the jobs where you can see every operations that have been built and can have a very detailed activity log on the operation and the time stamps and how long each operation took. For the Trace Explorer, it's limited to what the agents are doing. And so I think that's, what I showed here is that you do have, sorry. I'm moving bubbles away. Sorry. If I'm going to a specific time stamp, you can see, the time of beginning, the time of end of the operation. You can see the different variation and so on. For the agent, you will only see the the operation will be by each query. So each trace is the response to one question that you add. So conversation can be a set of trace. So in that case, we see that this query led to, sixteen seconds of processing and the other operation that we see in the screen right now. But other operation can be longer or shorter depending on how many calls need to be done. So and this trace view can be more complex if you want to have some specific trace that are logged in a different way. It's able to customize that, full code, as much as, as needed. I don't know if it's, answering the question. Can you let me know in the in the chat? Perfect. Well, if you have any other question yeah. So the question has been answered. Thank you, If you have any any further question, feel free to to to share them in the chats or in the q and a section. K. So we don't see any further questions. Last call maybe for a question, just in case someone is being shy. One thing I I forgot to to share and do that, in the times that we have is, what I I mentioned, like, and also Benoit mentioned, the be able to do low code and code aspects, and that's what we have this also this, separation with agents. The the they look different, but they're not so, so similar. So here we have an example of a visual agent. So in that case, it's an agent, where you just have visual configuration. So, like, every other visual recipes that we have, you can just select, LLM connectors that you want to use, and then you can select a set of tools based on the tools that you have built on your instance. That way, if you have a set of tools that is already prepared for your customers, for your, business users, so you can basically build their own a bit like Legos, agents, make sure to have proper, description, and tag them to the web interface. So that's, can be a basically zero code, process. If you want to go into more, detail and or if you have a more specific workflow, you have access to code tools. So here we have a full code agent. So it's in one interface, you will be basically building your own, interface. And so it's basically what I shared, in terms of behavior. So here is able we basically give the agents the ability to, sort a dataset for SQL queries to get a schema, get the description of the different dataset, do filtering, and so on. And, then you write, some form to say, oh, the agent is supposed to act and use those tools and plug that into building, building blocks, and that will be connected to other things. That way, you can have different level of interaction, and you can have hybrid behavior where we have a code agent that is becoming a code as a tool by a visual agent, so you can have, the level of connection and specifications that you need. It's also important in terms of, LLM because we have, now there's a lot of different elements, and so sometimes you want something, that is very smart that can make, like, complex decision. Sometimes you want something that is more straightforward and, very down to the point because and that can be, both a performance topic and also a cost topic. So you can put the money where you need to where you can pay for the bigger elements where you need to, and you can go for the simple one where you don't need to. That way you can manage, cost and efficiency in a more dynamic way. And just the last thing, here we see a set of tools. Tools themselves can be, managed as, visually. So here we have SQL tools that is basically just, connection between datasets. But this tool have been built by code, as a plug in inside of the table. So you still need someone to build the tool. Other tools are, already accessible, like, the data lookup will be a standard tool accessible inside of the table. Last thing I forgot. There was another question. So, yeah, there is one question about the access to the actual demo that you were showing, Anton, if if, someone wants to replicate the demo. So I guess someone being a customer. Yep. I don't think it's accessible right now. It, probably will be in a in a short term. Right now it's demo only. Product recommendation is a solution that is accessible for all, the TechWoo customer for free, and you can look at it look it up on the the TechWoo gallery if you want just to look for the business solution. We want the LLM aspects are built. But it's, it will be accessible as a demo, later on, but it's not yet. It's okay. And for for for Directo customers, I advise you to reach out to your, customer success, manager, and and we can definitely organize a follow-up on the topic, and and and, of course, keep you posted enough about the the release dates when, when this report and get, yes, general availability. Another question that we received is related to data governance. How is data governance implemented? For example, if a user, with a certain role is not allowed to see user ID or name in the answer from the chatbot, do we need to add the instruction in the prompt or through another method? I don't know exactly how, data segregation can be be performed. We can have different data set be being accessible for different holes. You can put the holes as you fit it to a specific agent. Using comms for data security can be a first layer, but it should not be, the only one. On this technical aspect, I don't want to to say anything wrong. So I'll I'll refer that to to someone else. But can you, how to say it, each LLM is built with a connection that has a set of walls, FTP with them. So you can manage who has access to which LLM connection, and so you can have control of if the process finding the agent has access to different datasets. So you can manage that for the access. I don't know if you can have something where the agent is able to glue the data some data and not share it. I don't think it's right now a safe process because you can still ask the agent to circumvent any prompt rules. So it should need to have a app coded behavior, so I don't think it can be under full strict prompt. So it's ever fully segregated or people have access to the data, but it can be managed partially with, sub agents like which I shared in the previous slides where we can have a specific agent that's physically managing, private information, but that's not a street barrier yet in terms of tools. But, the tech who comes with, managing, governance tools that a moment for when doing the structure and the architecture of the data. So if it's come if it come becomes an issue, it needs to be done at the design setup. It's not a perfect answer. Sorry. Great. Thank you, Quintin. I think there are no other questions. So thanks a lot for participating in this webinar. Thank you, Quintin, for the demo. Thanks for your questions. As mentioned, we will circulate the recording and the slides, later on, and and we wish you a very good day. Thanks a lot.