That is what I have been doing for the best part of 15 years in the ecommerce industry, and yeah today wanted to share some of the insights and some of the perhaps common questions we tend to come across when talking about workflows and optimisation workflows so to structure this I have been trying to follow little bit the week of a typical merchandising team. And we'll be going through the different steps that I tend to see with most of the organizations and then hopefully answer some of the common questions that tend to come up quite a bit when I work on optimization projects. So there's always a few questions around, especially along the lines of, so what is it, if we're using Threadable or Experience Orchestrator, what is it that we actually should change? When should we change it? How do we go about structuring the team? And how do we actually find out if it's worked? And so we'll be following these five steps here, which you'll probably find is something that you tend to follow on a weekly basis. Now, there are conveniently five steps, so I'm not necessarily saying this needs to be exactly weekly planning on Monday, the preparation on Tuesday and so on. But you'll find that these five steps tend to be spread out across the week. And generally speaking, we'll be going through these five things. So we've got the planning stage, when you're basically trying to define what is it that you're setting out to do in a week. Hopefully already looking a bit at measurements, the earlier the better. Then a preparation stage in which you do all the necessary work that's required to actually make those changes. So probably talking to your colleagues, looking at things that have happened before and so on. Then there's a bit about the rule and the strategy changes themselves. I'll not go too much into detail here, if that's okay. It's just because all of you are different. So every single retailer has different needs, has probably a different setup as well. So it would be too specific, I imagine, if I were to pick out something really specific here. So we'll be looking more at general patterns, what successful retailers change and how often they change it, but not so much the specific rules themselves. Then there's the evaluation step, of course. So after you've made some changes, have they actually worked? And how do we find out? And then last but not least, and sometimes overlooked, is the follow-up stage. And in basic terms, that's just a bit of housekeeping, really, and making sure that all of that hard work you've put in during that week doesn't go, well, doesn't get forgotten, really, over time. And yeah, that's the general stages we'll be covering in the next half an hour or so. So let's start with the planning stage. And this is probably something that will look really familiar because I'm sure every single one of your teams have been following similar patterns in their weekly setups. Also, I should probably preface this that I'm looking at a relatively merchandise-centric organization structure. If you're based in the UK or similar, you probably find this is the most likely organizational structure you're following. And I appreciate there's some differences culturally and geographically. So in Central Europe, for example, or continental Europe in general, the teams working with FEDEP tend to be a bit more product management-centric, but I'm sure the general patterns will still apply. So let's just have a quick look at how probably your week's already structured. So I'm sure you've all got a certain diary, a trading calendar that you tend to follow with some of the key dates in the year marked out. And so I imagine you will have some certain, especially sales that you basically, as a team, will always have to look after at a very specific date. And you know that's going to happen. You know that you have to look at it. You know you have to prepare for it. The really successful retailers I tend to work with, they then try to split the rest of the time. So those weeks that they're quite flexible with. into basically a mix of the typical BAU tasks, you know, all of the ones that tend to appear all the time and that are always super important and basically prevent you from doing something a bit more strategic. So what I've seen really successful retailers do is they basically set some time aside for strategic work. And strategic work in the sense, especially anything that goes beyond just, let's say, the typical daily tasks that you've got as a merchandising team. So this could be in a, let's say, in a Fredhopper context, this could be looking at search accuracy. So something that basically maybe looks at the more underlying concepts. It could be things like the ranking strategy that you want to set up or the recommendation strategies that you want to combine. And so not so much just individual products or individual product groups that you're pushing around, but more around what is this overall feature trying to achieve. And then the other thing that tends to be done a lot in successful retailers is an element of review and evaluation. So I've just marked this down here as the QBR bit. So the quarterly business review or a similar concept, depending on how exactly you work as a team, you work as a team, but just having the sort of element of looking back, have the, so first of all, have the strategic changes you set out to do, have they achieved what they were set out to do? Did they actually get done in the first place? But then also looking at the smaller changes, the weekly, the BAU changes, did they have any impact on the more underlying business-wide KPIs? And it's good to do that at least once a quarter. Again, just with the ability then to look back and hopefully take some learnings for the future. And at this point, when you've generally mapped out the different stages, you can then typically put some really specific tasks in there. So for example, maybe in your cell, you typically know what to do. But then for the more strategic time that you've set aside, maybe you mark out that in the first slot for that, we want to look at the new season ranking, which might come up a few weeks later. So there's probably some preparation time that needs to be done, possibly some testing, maybe even some development work that might be required. But it gives you enough time to prepare for it. So it doesn't come as a surprise when you actually get to the new season and the seasonal changeover. And you can do this for all of the slots. Again, I usually see this happening by weeks. So it's typically the quiet weeks that are set aside for the strategic work and the weeks that you know that are a bit more hectic for the more BAU tasks. And if you do that, you basically have a list of the tasks, a roadmap essentially, a trading roadmap. And as with all good roadmaps, you can apply KPIs to it to see if you're basically achieving the goals and also for the tasks themselves to see if those changes actually making sense. So for example, you could define, for example, for the sell-through, like smaller tasks, you essentially want to increase conversion. And then when you get to that point in April, imagine, you can just look at the changes setting out. You don't have to realign internally, maybe just to check that goal is still relevant. But overall, you know, you need to increase conversion. And you know, at the end of the week, you know how to measure it. Again, it's probably not pretty surprising, because at least for the BAU tasks, I'm sure you're following that process already. And then you can do the same thing for the more strategic tasks. And when you look back in the next quarterly review or similar, you can then see if you've actually had some positive impact there. And again, that's the sort of pattern I tend to see with many retailers. And then we go to the next step, which is the preparation of all of these individual changes. And here, as I've mentioned earlier, it's usually a combination of different topics that you need to do. So sometimes it's really just the basic tasks of getting a spreadsheet from a buying team or something as simple as that. Or for the more strategic work, you probably want to have a look at any kind of A-B tests or similar analysis, user testing, anything that's been done before that might help you with the decisions you're about to take. You might have to reach out to the right teams. So maybe you need some input from the buying teams. Maybe you need some input from brand marketing. So again, anything that's required from the wider audience is what you can use at this point. And of course, there's also the evaluation strategy. So this is not so much about how do you define the tasks that you want to do, but more how do you know if they'll actually work. So that's really good time at this point to think about how do you find out. Generally, we'll have a split here into the two different types of changes. So that's the strategic versus BAU on the previous slides. And generally speaking, the strategic changes you're making, by definition, they're broader. So they tend to apply to many pages at once, and probably also to a relatively long time period. So quite naturally, that lands itself to A-B testing. And especially when we're talking about relatively nuanced changes, let's say, for example, a ranking cocktail composition. So things that maybe from just looking at it, you wouldn't be able to tell immediately if it's right or wrong, especially if there's some conflicting goals. Let's say, for example, conversion rates versus stock management. So in those cases, you'll probably not be able to get around A-B testing as an evaluation method. And then on the other hand, we've got operational changes. So again, they're quite narrow, typically. So you often see teams working on really specific pages. So let's say, as the, maybe the merchandising team of a fashion retailer, they're looking at the dresses page, or they're looking at a specific landing or edit page. And I guess in the most extreme terms or example there, we might be looking at really pinning individual items. So for example, the brand market team has some idea that a certain product really needs to be promoted more. And because of that, the merchandising teams just have to move them around. It's a bit, well, a bit extreme, of course, but yeah, it still happens quite a lot in many different fashion retailers and not in other industries. So it's definitely one of the more common operational changes. And from an evaluation perspective, of course, that's, there can be a bit of a challenge. Just because if we're, for example, moving a single item on a single page, and possibly not for the reason of increasing conversion, but more around the, let's say, the brand marketing side, that doesn't leave us a lot of evaluation methods. So sometimes you just have to go with sanity checks. So essentially, did we fulfill the brand marketing task, essentially? As a statistician, of course, my, maybe that's not my preferred method, but there, sometimes there's no way around it. But of course, if possible, let's try again to use A-B testing. And sometimes we have to be a bit creative there. So sometimes we, maybe we wouldn't be able to check and evaluate a single rule, a single pin, for example, of a product. But usually there's not just a single one that you're changing. So there's typical patterns we can follow. And perhaps instead of testing if pushing product X to position one on a certain page is beneficial, maybe it's better to test if pushing products that fulfill some sort of criteria set out by the branding team, pushing them to the top of broad category pages, if that's a beneficial thing. So we can try to combine different tests, so different individual changes into a broader test, and then rather test the, let's say the strategy itself, as opposed to the individual rule. So just to give you an example of what I mean here is, let's say if this is a typical breakdown of a fashion retailer, the box size is essentially the sort of traffic that the individual pages receive. And the green box at the bottom, just as a visual representation of what the required sample size for successful A-B test should look like. And if there's, let's say a rule that's supposed to change the red dresses page, which is really specific, of course, there's no way we're going to have enough sample size for that test, especially if we're looking at a relatively short period, say a week or so. So we have to be creative. What we could do there is, we could say that, well, as you're testing the change for red dresses, maybe that's a bit too specific, but maybe we can combine all of the tests that change the pages for dresses with the color fields on. Because presumably, if you're changing that particular page for red dresses, you're probably using a similar idea of what products to pick for the black and blue dresses and all the others. So let's just combine them. And now we might have enough traffic already. Or the other option is to look over time. So if that change the red dresses page, as an example, is really so specific that we can't combine it with something else, then we need a longer time period. We should probably not include any periods that are really different. So for example, if the next week is a sales period, then maybe that's not going to work because you might have some completely different user behavior. So perhaps instead, let's look at the following week. And if you combine basically those two weeks, then hopefully we get to enough of a sample size. Just from a purely operational perspective, you'll probably need to set up separate tests to actually run this. And then that means you'll have to add up the numbers in the numbers in the Fredible interface or XO interface together and basically calculate the results there. But the more important thing is really to set up the tests and to think about the evaluation method at this point. Then the next thing at the preparation stage. So after you've got the general idea of what evaluation method is feasible, you should be able to do that. So if you're going to be able to do that, then you should pick a KPI. Again, no surprise here. It's probably also, well, it shouldn't be a surprise that conversion rate isn't the only metric. So sometimes, especially if you're again looking at relatively specific changes for which we maybe don't get the sample size, maybe looking at the basket rates might be an alternative. So that's why you can still get a good understanding of how close you are to the commercial outcome. But of course, you've got a lot more data that way. Next thing you have to look at, especially when you're picking KPIs for relatively complex scenarios, is to make sure that you're actually measuring the right thing. And this sounds trivial, but in practice, I've found this is something that's often done wrong, especially when we're not looking at the change that's purely there to drive conversion. So for example, I've had retailers try to increase margins in their product lists. So for example, they wanted to increase the percentage margin and added that as a contributing factor to ranking cocktail, then ran an A-B test and were disappointed that conversion rate went down. That shouldn't be really surprising because conversion rate is not the right metric at this point. Chances are that if you are looking at something that's really beneficial for a retailer, for example, margins, stock levels, even full price flags and so on, that's probably not the type of products that the user is necessarily interested in. So from a user's perspective, you might make the list a bit worse. But hopefully, if you're picking the right metric, the right KPI, you might see a positive impact. So at this point, if you're testing, should I include margin in my ranking cocktail, the KPI you're looking for is effectively the product profits, as opposed to just conversion rates. And finally, one trick that I've picked up over the years is to not just go with a single KPI, but define a measurement framework around it. So just have a basic, you've got one KPI that defines whether or not your test is successful. So if there's a significant change in that KPI, you probably have the answer you're looking for. However, to also add some additional metrics to it. So for example, if your KPI is conversion rate, and let's say this is a test for a search accuracy change. So of course, you'd first set out your hypothesis. So for example, by including more product data in my search engine, I expect my shoppers to see more relevant results. And then you basically define that how your user behavior should change. So they should be able to click more and then ultimately convert more. So your goal ultimately is more money. So conversions ultimately. However, if you see that your outcome in the conversion metric is completely different than the sort of expectation you set out in your user journey that you use in your hypotheses, then there might be something wrong with the test. So for example, if you see all of the other metrics go into the right direction. So let's say you get more clicks from search, you get more ads to basket. from search. But for some reason conversion rate goes down. Then that might be a case of, well, let's find out what's going on here. Because all of the it's going in all the right directions. But there might be just an issue with the measurement itself. Or there might be something else that that you maybe hadn't accounted for. But there's always a essentially a red flag at that point that you want to look into the test a bit more. Similarly, you might want to define essentially restrictions. So if you want to increase conversion rate. But the test outcome is neutral. So basically no significant difference. From a, I guess, from a purely statistical perspective, you could say that a test wasn't really successful. However, if there's been no significant drop in revenue, possibly not even a tendency, negative tendency in revenue, and all of the lower level metrics were positive, then there isn't really anything that should really stop you from making that change. You might not be able to put a number on it, at least not against the domain KPI. But you've probably made user behavior better if the click rates and add to basket rates went up, for example, and you didn't have a negative change in the revenue numbers, for example. So that's another way of how you can deal with the tests themselves to avoid basically wasting the test slots. So this here could be an example of how to break down the KPIs into low-level metrics and then have these other restrictions on the side. So for many of you, this would indeed be, let's say, if you're looking at increasing conversion rate, then you probably don't want to have a negative change in revenue. You maybe want to keep an eye on the markdowns of products so that you're not pushing users towards the markdown products instead, which might have a negative impact on the profit ratios. And then we come to the changes themselves. So as I mentioned earlier in the deck, I'm not going to talk about specific changes, but more about the type of changes we tend to see. So if we're looking at a typical Fretiber customer, I've tried to summarize a little bit how I often see retailers make those changes and how often that happens. You'll see here that there's a direct correlation to the type of rules that are perhaps more strategic versus those that are a bit more BAU. So what I typically see is that essentially those changes that really happen on a, let's say, weekly basis, if your organization uses that feature for CDI BAU tasks, is to use result modifications for, for example, promoting products that are currently highlighted in the campaign. So if you're campaigning in the marketing campaign sense, not in the Fretiber campaign. So for example, if you're running an edit, a marketing campaign at the moment, and there's a few products that are simply highlighted basically in all of the messaging, then it's probably a good idea to use modifications. So for that to also align the messaging on site. Unless there's a good data point for it, you'll probably be using result modifications as something relatively manual. And that does mean that's, yeah, it needs to be updated. You'll probably also looking at making those changes quite often if you're using result modifications for search results, for example, just to fix like a search here and there. It's not something that you would change for the same search week over week, but it's just something that if you come across some, like an issue, well, add a synonym here, add a result modification there, perhaps a redirect here. So those, those changes tend to happen quite a lot. Again, usually not for the same type of, of searches or pages, but yeah, there, there tend to be a few of those changes. And then we've got the, I'll jump to the, the right side here first. We've got the, the, the, the, the, the, the, the relatively rare changes and those tend to be more strategic really. So the, the classic example, of course, is the, the search algorithms. So those, those changes that are, that are essentially hidden in the deep in the system settings. Typically they're, they're only changed as part of a project. So I guess by definition that makes them quite rare. And then the, the other thing that, that I usually like to see a change quite rarely is the ranking rules. So especially adding them. So that's, there might be a bit of a surprise here, but I typically see ranking rules as something that's, that shouldn't really change that much. And that's just because typically good retailers have a ranking rules for, for specific strategies. So you, you wouldn't normally need to, to change them that often. Cause you've probably got one for you, for your regular pages. You've got one for your sale pages, for your new in pages, perhaps a few other scenarios like this, but you, you typically wouldn't set up a, let's say a new ranking rule just for the, say red dresses. Whenever I see something like this, it's usually a sign that's there's something else underneath that's, that isn't quite right. And it's, it's essentially just a symptom that maybe something needs to be looked at. And then we've got the, the area in between. So those, it can be a bit of a, of a gray area, depending on how you operate with the different system. But typically those rules, they tend to change from, let's say maybe monthly to, to every, every quarter or something like that. And typically there's the, like changes that, that either are seasonal or that, that are changed as part of an ongoing optimization. So to, to give you some examples, the, the ranking characters, I guess is the classic one. So you, you might have the, for example, a, a, a bit of ranking logic that, that caters, especially for, sale periods. And so the, the moment you get to a sale period, you probably want to enable that either in a, in a cocktail or in a ranking rule, but you, there's probably a small change you might have to, to make. Um, just to, to basically tell the system that we are about to go into the, the sale season, or maybe we go into a new season and we, we want to use the, the changeover rule now. Um, similarly for, for result modifications that are, that are not, um, let's say pinning individual products, but rather that are, that are adding structure to your product lists. So those, again, they, you shouldn't have to change them too often. So if you see yourself changing them on a weekly basis, um, that's probably a bit too much, uh, but, um, seasonally changing them is something that's, that's relatively common. Again, just to, to account for maybe different trends that you want to promote. Um, sometimes even just the, the type of products that are more, yeah, that are more suitable. for the, for the time of year. Um, same for, for, uh, recommendations, of course. So there are, there are similar considerations as for the, the rankings, the, the structuring. And finally, the, the filters and facets. There's not so much because it's an optimization strategy there, but, uh, it's just, uh, yeah, there's only so much you can change in, in your filters or should change in your filters all the time, uh, to, to not confuse customers too much. So the, the, the filters shouldn't really, uh, jump around all the time. Um, cause that, that would be, or could be a really confusing experience for your, for your shoppers. And as you can, as you can probably see here between the lines, um, the, the changes on towards the right side of the screen, they all tend to be a lot more dynamic, a lot more strategic, and most importantly require a lot less effort. So if you, if you, if you're working as a team, um, um, you probably want to, to use your preparation, uh, time in the, in the, in the week, or perhaps some of those slots for strategic changes to also look at the efficiency. So maybe if you find yourself, uh, making the, the, sort of, uh, changes on the left-hand side, maybe, uh, you find some ways how you can group your, let's say, the, those weekly changes to the result modifications. Maybe there's a way of, of actually improving the, the search configuration to reduce the need for synonyms and result modifications and redirects. There's, there's usually some options. And if you're interested in that, by the way, just reach out to your customer success manager, um, or to myself. And yeah, we, we can perhaps think about some options there, but generally speaking, you really want to go towards the, the, the more dynamic rules, those that you, that you only have to look at once in a while. Um, which I guess brings us now to the, the evaluation stage. Um, if we've done the preparation step, uh, well enough, then this, sorry, this state is, is relatively straightforward. Um, again, depends a little bit on the, the exact type of changes you're making, but generally speaking, if you've, if you've prepared the evaluation method, you just need to execute it, look at the numbers, um, basically draw a conclusion from it. And, uh, yeah, depending on what that conclusion is, we'd now go to the, the followup stage. Um, hopefully everything's been really positive and you can just document your successes. Well, make the changes first and then document, um, basically what you've tested, why you've tested it, why you've made a change in the first place. And, uh, and, uh, and this is something I've really learned. And sometimes really just write down the, the exact changes you've made. So take a screen grab, uh, put it into your Confluence page or Word document, whatever you use to, to document your changes. So, because in, well, six months down the line, uh, you might remember the test, but you'll probably not remember the exact setup. So it's, it's always good to just write this down. Um, um, generally you want to, to use your quarterly or similarly, uh, reviews to just make sure that the documentation is up to date and possibly also to, to look a little bit back, um, into not just the, the current quarter, but also back if there's anything similar that, uh, that maybe jumps out, which then can help you, um, decide on the, the next steps. So maybe there's, there's a AB test that's, that's had a bit of a, of a funny result. And you, maybe you, you, at that point you remember that maybe we should do a follow-up so that can help you, um, to find some of the, the hypotheses and tests for the, for the coming quarter. And then probably the, the most overlooked of the, of the steps is housekeeping. So, um, especially if you're making many, many changes, so that this is often in case, uh, you're doing the, the more manual type of changes. Um, you don't want to leave all of those rules sitting around from a, from a technical perspective, unless you're adding thousands of rules, it's probably not going to have too much of a, of an impact on performance, but it makes your team a lot less efficient. Um, just from, from personal experience, it's, it's really hard to go through thousands of rules that all aren't really up to date anymore. And then to, to work out what's exactly, um, leading to, to a certain result. So it's much better to, to just, once you're done with a, either a test or a certain change or the, the period of time that's, that a rule was set up for has passed, just at the, the very least use labels, uh, to mark them. Mark the ones that you want to review and to, to delete, or ideally just delete them. If you, if you don't think you'll need them anymore. And, um, the, I guess the best teams are, I tend to work with, uh, depending on how, how manual the type of changes are that's, that they're, that they're making, um, cadence there's typically weekly to at least once every, every few months or so to, to really look at all of the old rules that need tidying up. And if you do use labels, um, in the federal merchandising studio, for example, it really speeds things up quite a bit. So my, my personal trick there is to, to just mark the, let's see if there's any rule that, that I know will not be applicable for the rest of the year, but perhaps it's only applicable for the, the month of March, perhaps just add a little label, um, clean up in April. And then come April, uh, just select everything that's marked with April, uh, go through the list and hopefully be able to throw out everything that's no longer required. And as, uh, at the point when you set up the rules, that's, that's only just a few seconds, but it really saves your time massively down the line. And yeah, that's. Should bring us to the end of the, the week. So hopefully, uh, then a nice weekend and we can continue with the same sort of process. the week after. So maybe just to summarize then, uh, quickly. The, the most important thing is to, to really be, um, to plan ahead, to think strategically and to essentially set time aside to make those more strategic changes. So it's so easy to get overwhelmed by the, by the small weekly tasks, by the BOU tasks, by the firefighting that we all have to go through. that there simply isn't any time to, to think more about the, the sort of work that can help you either increase the, the KPIs that you set out to do or to help you reduce the, the workload you have to go through. So really set away, uh, set aside a few weeks, um, that you really want to work on those, the small street, uh, strategic bits. Then a measurement, of course. Um, you've heard me talk about the, the methodologies and the evaluation methods. So, uh, as early as possible. So this is really, it's quite critical. Um, if the, if you only look at the, the measurement after the fact, you'll typically come across the, the, the scenario that you then have to, what to make do with some, some methodology that's not really as reliable as it could be. Um, you'll possibly come to a conclusion that you, well, you don't know if it's actually worked. Well, if you look at the, the methodology as early as you can, then, uh, you can typically make small adjustments to the, to the rules, uh, so that they actually become more measurable. Um, at the same time, there are just a few changes that maybe, maybe you can't really measure. Um, yeah, as much as it pains me to say that, but, uh, sometimes this is just the way it is. So if there's a, if there's a, let's say a business need to, to do something and, um, yeah, let's say the, the outcome of a test wouldn't really change the, the way how you do things. Maybe it doesn't need to be tested. Um, next on the, the methodology, if you're looking at really specific changes, don't necessarily test every single individual change, but try to group them wherever you can. Try to think about, um, how are you making those changes to the rules? Why are you making them? What is it that they're actually trying to achieve? And then that helps you define groups of tests, which increased sample size, which is really significantly reduced the, the amount of time that it takes to, to actually come to good conclusion, whether or not those changes are useful. And finally, housekeeping. So the somewhat boring bits, but they, they really help you down the line and they will save you so much time. If you, if you get into the habit of, uh, keeping track of the, the rules that, that, that's, that need to be cleared out at, at some point. And yes, uh, that's the, uh, type of changes I tend to see. And yeah, thank you for attending. And if you've got any questions, there's a Q and A box and I'd love to answer your questions. Thank you. Any last chance to, to ask questions. Ah, we've, we've got one question here. Thank you. Um, what are the, the KPIs that you recommend focusing on, uh, uh, for merchandising success? So that is, yeah, it's a good question. Um, I guess the, the answer it will have to be, it depends. Um, so depending on the, the type of retailer work with, um, yeah, everyone's got their own little ideas of, uh, what KPIs should be used, but generally speaking, it tends to be conversion rate of revenue across the board. Um, I, so there's there's, I guess, from a, from a statistical perspective, conversion rate is a little bit easier to work with. Um, it just requires less of a sample size, but then if you're, if your goal is to increase revenue, then maybe it's worth, uh, having a slightly, uh, larger sample size and then really try to get an answer for the, for the impact on revenue. Um, no, that being said, as, as I mentioned earlier, that there's sometimes, uh, a case for, for using, lower level metrics, uh, if that helps you speed up the, the AB test, especially if you're looking for some really specific changes. So in that case, um, you might be able to, to, to either use, um, something that is, let's say, a basket, uh, something. So in other words, something that's close enough to commercial outcome. Um, but, and also at the same time, still fully within control of the, let's say the, the Fredipa solution or the XO solution. Um, so yeah, it's the basket is, is a good one. I wouldn't necessarily go, uh, lower than that. Uh, just because the, yeah, if you, for example, um, use click rate as a, as a KPI, there, there may be too many other things that, that have an impact on it. Um, so yeah, I guess, uh, using lower level KPIs is, is one thing that I've, I tend to see sometimes. Um, if you just need an answer relatively quickly, or at least, uh, uh, a reasonable guess. So admittedly it's the, probably the ultimately the conversion rate that you'd need to look at or do the revenue. Um, the other thing you can do is, uh, to, to filter, um, your sessions into, into those that are really, um, that are affected by, by the changes you're making. So for example, if you're, if you're testing search changes, um, yeah, you don't, there's no need to, to look at sessions without search really. So that again, that might reduce the, the noise in the background and hopefully get you an answer more quickly. Um, then another question, uh, another one about AB testing, um, how frequently should we review and document AB tests to ensure we are learning from our strategies? So here, um, the, I guess it depends a little bit on the, the sample size ultimately. So if you, if you're receiving lots of, of traffic, um, and you're running really broad tests, typically two weeks should be enough to, to get you a sick config answer. So always consult a, an AB testing sample size calculator, of course, but, uh, yeah, for a really broad test, um, it often is, it's two weeks. Um, again, as a statistician, I would not look at the, the results before that, before you reach the sample size. There's, uh, it's a bit of a, of a bad habit to be honest, um, that you, you try to draw conclusions from tests that haven't finished because they can sway the other way around after, uh, receiving more traffic. Um, so yeah, but basically the short answer is, uh, wait until you've reached your sample size, uh, the one that you've calculated before, hopefully, and then, then look at the results and, um, draw conclusions. And I would try to document the, the findings as quickly as you can. Uh, we've had one question here, um, on your trading calendar, um, what does BAU stand for? Sorry, I should probably have, have at least defined it once, uh, business as usual. So, uh, in other words, just the, the small tasks we all do on a weekly basis or however frequently we work on, on the, the sort of, uh, fretable or XO changes. Um, one more, uh, how can we balance strategic work with the day-to-day operational changes that need to be made? Um, yeah, I have to admit that, um, when I say setting aside time for, for strategic work, uh, uh, it's probably a bit of an optimistic view. Um, because we, we all know that, uh, every single time we set some time aside for something important, uh, or something important long-term, something really critical, urgent comes up in the short term. Um, but I guess, yeah, the, the only thing there is to, to try to set the time aside. And yes, you, there, there will always be a few things that come up, um, but try to, to really prioritize this strategic work. So unless it's, uh, something short term, that's really absolutely critical, maybe, um, try to stick to the, to the original plan. Um, and I was easier said than done. And the, I guess the, the other thing you can try to do is, uh, to maybe even if you've got a, a larger team, so perhaps you can, you can even split the, how the team work. So maybe if, if half of your team, um, in the first week, is looking at the, the more strategic things and the, the, the second half of the team is looking at the, the, like the more daily changes and then the, the week after changes. So maybe that's, that's also a way how you can deal with the, the interruptions that you tend to get so that the, the, basically firefighting goes to the, the part of the team that's already looking at the, the, the more day-to-day tasks. Do we have any other questions? Okay. Okay. Then, well, thank you all for attending. And as Christina said, uh, we'll be, be sharing the, the recording shortly. Thank you very much. Goodbye.