Transcript
I'm Summer Simpson. I'm a VP of product for our data product at Amplitude, and I've been doing the interwebs for about 30 years now. Um, Alex, like. intro yourself. Yes. Sure. Thanks. Yeah. So I'm Alex. I've been working in product analytics for a few years now. My background is originally in finance analysis and forecasting, but as I started working through more and more forecasts, I just found that the user behavior was something that was ending up being a precursor for forecasting and obviously analyzing and forecasting user behavior sort of dropped me into product analytics, as you will. Yeah, that's how I kind of started off. Full. Uh, interesting journeys, how we, how we get to kind of where we are. I mean, I started out in journalism and here I am in digital analytics, so numbers are important, right? Yep. Yeah, exactly. In all of life. Yeah. Yeah. So let's, let's start off. I want to I want to hear a bit about Brainly. Yeah, sure. So for those on the call who don't know what Brainly is, it's a web and app company all across the world helping kids with their homework. So we see around about 300 million users, school kids every single month going on asking questions, mostly using the Q&A forum that we have on there to get the answers to the homework that they have. So it's very much a back and forth between different students. The product offering has then expanded drastically. We have sort of tutoring services, instant answer math solvers, all of these kinds of things trying to enhance with ML and AI as well. So it's been a really interesting journey. We've definitely expanded very, very rapidly, so it's really exciting to see us being one of the number one education apps really out there in the world, especially for school kids. Excellent. Thanks. Um, so let's just jump kind of straight into it. Um, you know, when it comes to, um, you know, getting the information and stuff that you need, I'm really interested in digging into some of the, the problems and challenges that, you know, you've, you were looking to solve and maybe still are looking to solve. Um, and kind of how you measure success and the value that you get from Amplitude. Sure. Yeah, sure. So Amplitude has been a great help for us in the last couple of years really. Uh, after we implemented, we really saw that we were able to understand a lot of those funnels and user journeys that we couldn't really get to grips with beforehand. And that was really something that was very important to us. It was the flow from users going from web and moving across onto app. So trying to understand that attribution and how they were moving across onboarding flows, making sure we were capturing the users at the right moment, also the product usage features. So I mentioned there that we were sort of expanded our product feature range, one of the ones there being instant answers and using the power of Amplitude, we were able to see the correlation and then test out to then find the causation between users being onboarded and seeing that feature very, very early on and then ending up actually in a trial and then a subscription. So our measure of success, obviously we have the usual product usage stats, your retention metrics, all of those kinds of things. But we do obviously boil down to a subscription model. So testing out reverse trials, free trials, the actual number of signups, etcetera, etcetera. That's kind of where the success comes to and being able to use that path and understand the correlation between all these different behaviors. And then the build up to the actual subscriptions has been fantastic, especially in the last year, I would say excellent. And yeah, so you're looking at basically solving for that that product growth loop, right? How do you do you, do you fit like experimentation a B testing multivariate into that? Yeah, yeah, definitely. So we've actually recently onboarded Amplitude experiment as our platform for experimentation in in the last six months or so, we've seen a really big uptick actually the frequency and the speed that we're running our experiments now on Amplitude experiment has been fantastic. We we've actually just implemented, for example, some reverse trials and different subscription plans, etcetera, etcetera. So that's been wonderful to be able to see and run through Amplitude experiment and then cleanly feed through into Amplitude analytics. So then you can get that whole broad picture. So we're not just now looking at the experiment itself, we're also looking at the future behavior of those users, because especially when you're looking at things like subscriptions, which is a very lagging sort of metric when you're trying to draw users towards subscriptions, it might be very early on in the funnel. So running that experiment is then nice to be able to track that all the way through and then trick all the users through in their journey to see if they do end up actually signing up and then also retaining that subscription as well. Cool. Yeah. As a, um, you know, with, with my product teams, you know, it's always about like, we got to move fast, right? We got to like, get enough information to be able to like, make big bets and decisions and stuff like that. How does, how do the tools that, that, that you've been leveraging kind of help you do that fast and iteration and you know fail fast learn? Yeah, yeah, definitely. I think the the way that the I wouldn't say that the analysis is instant and can be seen straight away because we do try to make sure that it stays behind a wall whilst you're actually running the experiment. That's one of the things that is always difficult with stakeholder management is when the product owner or the product manager comes in and says like, Well, how's the experiment going? After two days, you kind of need to kind of keep them calm, keep them calm. But the use of things like dashboard templates has been fantastic. So we've implemented things like guardrail metrics that we are running in all of our experiments. And instead of having to run that analysis through every single time we run an experiment and there's multiple ones live every single time, you're just throwing that into that Amplitude dashboard, changing the properties so that you're looking at that specific experiment, and then you can instantly see whether or not you're harming any of your guardrail metrics, for example. And that definitely speeds us up. Excellent. Excellent. So your your expertise in this area led you to found Magnuson Analytics. And you talk a little bit about that journey. Yeah, sure. So Magnus Analytica actually started off as a structure to work with Brainly getting this one huge, huge client, really amazing in the educational space to be able to kind of work with them. And we just saw that the need in the market just expanded and grew like crazy. Especially we have quite a lot of clients across Europe, but the main market space that we're going through is those startups. Anyone that's sort of on the free plan of Amplitude that don't get so much of the support from the professional services team within Amplitude. So we're seeing a lot of clients coming directly to us, in fact asking us to support with either their Amplitude implementation analysis, audits, maintenance, etcetera, etcetera, kind of working with third party vendors and tools like all the integrations obviously Amplitude has. So we're working with revenue or braze or appsflyer or whoever that might be. So that just absolutely exploded. I would say over the last year, year and a half we've seen that real need as the uptick of clients who are using Amplitude has obviously grown and Amplitude has been doing amazingly with its client portfolio recently. And it just gives opportunities for companies like like ours to just come in and assist those those teams with their analytics and really drive the culture. That's great. Um, you know, it's interesting. Like even like talking to Amplitude customers, I find that, um, you know, people are at different places when it comes to like the maturity curve. And I kind of see things falling into, into two different buckets right there. They're focused on, you know, how do I, how do I set up my tech stack, how do I integrate this, you know, and the technical problems. And then there's the how do I get my internal teams and other functions to buy into it? How do I manage that cultural shift? Can you talk a little bit about that and like your your client base and some of the challenges that they they run into? Yeah. Yeah, definitely. So. So the client base is pretty wide, very varied all over the world. Those in the educational space, those in finance, those in e-commerce, all different sizes of companies as well, from those that are just like to people running it on their own. Um, kind of bootstrapped themselves all the way up to hundreds or thousands of employees, etcetera. And we see the same sort of problems occurring depending on the type of client that we're taking on. So those who have already got their implementation done, they feel like the technical side is there. And so now they're trying to get that cultural shift like you're mentioning. But generally when we come in, they say, well, the events are there, so now can you do your thing and create some charts for us and make it all cool and give us a fancy looking funnel and blah blah blah blah, blah, and you have to kind of slow them down, go through a very quick audit. How did the implementation actually go? And generally you do see some points that need to be improved there before you can then move on to the cultural side of things. I find that it goes hand in hand almost. If you get the technical side and the implementation done really nicely, really cleanly, your taxonomy document makes sense. It's maintained. You don't have errors coming through, you're tracking what needs to be tracked. Then it's so easy to use Amplitude. I mean, that's the beauty of Amplitude is that anyone can use it. There's drag and drop interface. Everything has been worked out. I can tell by your team that it you've done so much research into how users are using the product that it's pretty hard to go wrong generally. But one of the main roadblocks there is that implementation at the beginning. So I see a lot of clients saying, Well, can't you give us this chart, Can't you do this analysis? And we have to go back to them unfortunately, and say, well, you need to change your implementation. We need to talk to talk with the developers, we need to change these events, etcetera. Um, so that can definitely stunt the growth. And I think that's where that cultural part, um, you'll find, find the issues with a lot of other clients that we've come into that are completely new to Amplitude. They may be migrating across from GA onto Amplitude and they want someone to actually do the implementation for them. That's a that's a great situation to be in because you are designing the taxonomy, you're making sure it can scale. That's a really important thing. Some of the clients that we've seen that have done their implementations, they've done one project based on web and another project based for their app platform, and then they're like, Yeah, but my, my users jump from the web onto the app and then back again within the space of a day. Well, don't put them in separate project spaces, put them all together. So some of these things that can crop up, we can just avoid completely if it's a clean from scratch implementation and that just you just see the speed and the pick up of their teams in how they want to use Amplitude because it is so easy to use and they do get those insights in minutes instead of days, pretty much. Um, so yeah, that's where the technical side of things can come in. Definitely. Yeah. Um, one of the things I found too is that people think, okay, we're going to, we're going to do our instrumentation, we're going to like, hook up our sources, and then that data is just going to like flow in and then boom, magically in the charts. Like, um, what's, what's, what are people's kind of reaction when you're like, Well, but wait, you, you need to like, you need to plan, you need to govern the data. Um, how do, how do people kind of do they figure it out on their own or is that something that you have to kind of help them with? Uh, generally, we do have to help them. They there's this switch that goes in their head. We get kind of the same sort of conversations going each time. And when you do say that, that specific thing that you wanted can't be done until we change the implementation because the implementation was I wouldn't say done wrong, but it wasn't done right, then suddenly they're like, Oh, well, yeah, I guess that makes sense. Like you need to have a good foundation in order to build a beautiful house. So now we're talking about the foundation and then that's when it starts to click with them. You do get a couple of clients that are a little bit, uh, they push back and say, Well, the tracking is all there, all of the events are coming through, so why can't you just clean it up and make it work with the volumes of data that we're obviously playing with and depending on sort of like Amplitude plans that they're on, etcetera, it can be quite difficult obviously, to then manipulate the data to make it clean, to then work with it. And there's obviously the very famous phrase that I won't say live about what comes in and then what comes out. Um, but there's only so much that you can change it in the middle. And a lot of the clients do realize that in the end the painful bit is then when you try and tell them that they have to probably start from scratch depending on how much needs to be changed. Yeah, yeah. It's, it's funny. It's like in that, in the cultural sort of evolution of, um, you know, a company, um, you know, people after they've gone through this, they learned that, you know, it's product analytics, it's a team sport, right? It takes a, it takes a village to get good insights out of good clean data. Um, you know, product has their role, engineering has their role data analysts have their role. Like, um, how, uh, how quickly after people sort of like understand that governance step, do they kind of get to that, that ultimate democratization phase? That that was a crazy shift for us, actually, very, very recently. Um, so at Magnus Analytica, we onboarded someone specifically for data governance. So I have a bit of a background I can get by with clients, but it was really nice to get someone who was really specialized in it and they were just looking at that day to day and the uptick in the usage stats. So your weekly learners that you would have as your kind of North Star metric at Amplitude, we were tracking that sort of dashboard with each of our clients and from the date when we hired the data governance specialist back in October, it took a month and then there was just this skyrocket in the usage from from our clients, the number of members of their team that were going in and actually using the data because there were good definitions, there was good categorization, there was good tagging, there weren't errors in the data. There wasn't properties that should be populated coming out blank for, for example, which like when someone goes in and is trying to create a chart and then they see that it's blank on the group, by that they need to do quite often they'll be giving up, whereas it just it just jumps within a month and it just stayed there and it was lovely to see. So yeah, definitely one of the best moves that we ever did was concentrating more on data governance. Yeah, that's cool. Um, it's definitely an important step. I mean, and I'm always, always asking the team and wondering it's like, what, um, what can we do to help people, you know, as, as software developers and, and product owners help people realize the necessity of that step. I mean, I feel like some days I'm like, should we just start handing out, like, Starbucks gift cards to get people to come in and do some governance? Like what? What do you find is sort of like a, um, a good trigger, right, to get to get folks doing that? I think your data platform is brilliant. I think one of the main things that we saw that was a big push back for everyone was having to deal with their taxonomy document in Excel and people hated that, like just building up an Excel taxonomy document and then you couldn't really do any data checks on it anyway because it was just a static document that wasn't connected to anything. So having a beautiful UI to interact with that, you can then collaborate with other people on and then the data feeds through live and you get those automatic checks. That was great. And then very, very recently you've added in the data health summary on the home page. If you send a screenshot of that to your main data owners once a week, once a fortnight and just say like, Hey, reminder we have 15 Uncategorized events and 18 events with errors, they're jumping in there like instantly, like within the hour of sending that little screenshot, they're jumping into data Amplitude. The data platform and having a look at like what events are those like, how can we fix that, etcetera. So yeah, I think that what you've done already has been a really great leap forwards and it's just that ease of access for people to be able to go on to that platform, to be able to understand how it works, to be able to make adjustments to the events that are being tracked and ensuring that the descriptions are included. One of the small things that I would I would add in that data platform is the screenshot mode, or you can do it when you're looking on the analytics platform. But I all the clients I've worked with that have really, really high uptake in Amplitude. They love seeing when they're going down, creating a chart and they see an event and then the screenshot pops up of what that event is firing with. So being able to do that directly in the platform would be really great. But yeah, it's that ease of access. Cool. Very cool. Yeah. So like gamifying it for me it would be just putting red circles with numbers in it every everywhere because it's like I'm obsessed on my phone with clearing those out, so. Yeah, yeah. Um, well, let's switch gears a little bit and go a little bit more tech. I'm, I'm just I'm dangerous enough when it comes to, to the tech side of things to annoy my engineers. But so, you know, as a as a data leader, how are you how are you looking at maintaining the cost of your of your stack. Right. And making, you know, while you're making your teams more data driven. Yeah. So first things first. Not just the costs, but right at the very end you've got the ROI. So what is your return on investment? So I've seen a lot of clients, especially the smaller ones, that do want to save costs. They try to go for the cheap option and then they don't end up using it. And then it's just that extra cost that's gathering dust in the corner. So a little bit more of that shift towards the best of breed always does help. And then when those best in breeds really integrate together very well, that's fantastic. If you have multiple different platforms that don't talk with each other and then you have to either bring in a completely separate CDP or you have to build specialist pipelines and data flows between all of those platforms. That's just extra cost, that's extra data ops, extra data engineers, extra software licenses that you're paying for. So if you get those that work seamlessly together, I think that's fantastic. One of the ones that I see cropping up a lot often called the holy trinity of product analytics. It is that Amplitude braze Snowflake connection like the three of those work so nicely together that you just open the three of them up, you just connect them and it just works, which I find fantastic. And one of the things that we're working on very hard at Brainly at the moment is the bi directional flow that you have with Snowflake, which for us offers a whole new level of of analytics and of maintenance. So we can send in super, super clean data into Amplitude if we want to bolster it with our back end data. But then also our entire event stream is coming back out into Snowflake so that the analysts can query it so that they can join it onto some of the back end data that isn't already being fed into Amplitude. And also, as great as the Amplitude is, there are other specific tools that are made for data visualization like Tableau and just connecting from Snowflake into Tableau to really do that advanced analysis and that advanced visualization, that whole flow is just like really clean and super efficient in my mind. Excellent. Um, it's funny, like, you know, just how, how much stuff has changed and hasn't changed over the last 30 years since the days of counting hits. Um, I remember those, uh, you know, you've got, you know, ETL, reverse ETL, no ETL, you know, all kinds of things happening and working with, with, with customers and helping them in terms of their of their stack. Are there any sort of trends that you're kind of you're seeing as of as of late when in the past like, you know, year. The counting hits thing like you say it's definitely still here obviously Amplitude is moving away from from the event volume subscriptions onto the monthly tracked users but you still see a lot of those subscription models that are around usage or hits etcetera. So we we look to try to optimize as much of the flow as possible. So that is looking at what data you need and what data you don't need. And data that lands in Amplitude normally is of one of two aspects. It's either the actual product analytics, which is what you want or it's operational analytics. We find that a lot of our clients are just capturing everything. And when I say operational analytics, I mean in the things that your developers want, what the engineers need, the crashes, all these kinds of things that product analytics itself you don't need, you still want it, but you just don't need it in Amplitude. And so the cost of flowing all that data through, storing it in different warehouses or whatever, reengineering it, sending it back through, we try and split off as much of that as possible. So the operational side of things, the operational hits, they just get stored in one huge unstructured data lake and then you can do whatever you want with it and it's pretty cheap. And then the more important product analytics, that's where we'll do the cleaning, the refining, the visualization, the live updates to the dashboards in Tableau, for example, those things that will cost a little bit more when it comes to the usage that then gets siphoned off. So you're definitely trying to look at that optimization. So when it comes to trends, I would say the consistent one is that hits and product usage and then the optimization is a key one, especially with how the economy has been going at the moment. I think everyone is optimizing everything and it's the same with data as well. It's the size, it's the amount, it's the time that you're spending on it as well. Now what you mentioned about the the engineering like type events, you don't hear that use case come up very often. And it's an interesting one, right? Like being able to monitor and track track. Like 404 or 503 or some sort of error on the back end. And have you found anyone who has like any of your customers have done a really good job of tying that information in with the product analytics and looking for some some causal? You know, obviously, if a if a user is going to run into problems. Right. That might impact their their long term longevity with the with their product. Have you found anybody that's done that well? Yeah, sure. So we we've had a couple of clients that have done it in different ways. Some that are a little bit smaller, that have just sent it through to Amplitude and it just worked. And one of the beautiful things when it comes to Amplitude with things like errors is using the Pathfinder. That's fantastic. Having a look at sessions or journeys that end in an error screen and then seeing the Sankey diagram flow back from that to see where the errors are coming from, it really helped. Then the dev team to start to prioritize where their bugs were coming from so that then there wasn't quite as much of the QA that they needed to do. Obviously they were trying to cure everything before the features were being released, but some things just get missed and then that was like a really great way for them to to see that coming up. So that's what we see from some of the smaller like lower volume clients. And then the ones that are higher volume, that's when the unstructured data lake comes into effect where you store it all in Snowflake, you get all of your errors coming through with information about which which user it was and what session they were in. And then your Amplitude data is flowing through into a structured table within Snowflake, and then you're working on building the two of them together and then running a report through something, for example, Tableau or Power BI or whatever it might be. So then you can start to get that as a generally like a weekly report refresh through so that you can just turn the wheel on that. And then again you'll see which sessions are leading up and causing these errors to come through. That's been really interesting to see. We've only seen it in a few clients, something that I'd like to do a little bit more with others. Yeah, definitely think that would speed up the engineering time. Yeah. So my, my, my Nirvana, right, is being able to take all the data to understand the customer. I mean, from your standard product analytics to those types of like the error metrics to, you know, pulling in data from your support systems, right? And so you've got that complete sort of 360 view. Um, yeah, I kind of think that's everybody's Holy Grail though, right? Yeah, yeah, definitely. And I think the easiest way to store all of that would be in Snowflake personally, just because of the way that you can then change and adjust and re-engineer things. You just get such a wealth of data that can come through from Amplitude, a huge, huge Json files with all of the different user properties and event properties for every single event, every single session coming through, and then just connecting that up with the other integrations that you've got from your other platforms. Like you were saying, the sort of support data that you've got. Yeah. Nice. All right. So there's there's one thing I always ask every, every customer I ever talk to. So you said a lot of really great things about Amplitude and I appreciate it, especially like our our product area. But what is what's one thing that that you would you would ask us like is there something we've missed? Is there an improvement we could make? What is what's something that we could do to make your life and your your customers lives better? Oh, it's a tricky one because so I talk with the customer success teams like very frequently, and when there is something that comes up, it generally gets fixed within, I would say 3 to 6 months. It's been fantastic. The kind of turnover that you do have when it comes to bugs or product implementations of new features at the moment, what have we got at the moment? So I mentioned around the screenshotting in data, but that's like a nice to have. That wouldn't be something that I find is an issue these days. I've had a couple of clients that are very old school that like their brand colors coming into charts. Now I don't personally agree with it, but sometimes when you have a client that is like, I need it done because I'm an old insurance company and I have a board who needs to have this deck, or maybe it's investors that need to have a proper pitch deck coming through with my brand colors, not with Amplitude colors. So the fact that you've got I think you've got three color palettes at the moment that you can switch between in Amplitude, there's the, the basic one, there's the pastel one and then there's the pride colors. I think being able to give the client the opportunity to add just one more palette would be quite nice. And they can just add in like 6 or 8 colors, however many it is that are their brands and then they just switch that on for either their organization or for specific users and they can just make those pitch decks as and how they need them. Again, I'm not saying that it's great to use specific brand colors. There's a lot of people that say that that's a really bad, bad thing to do. But yeah, it's a nice option for especially some of the old school clients that we have. Very cool. I like that idea. Um, we'll, we'll drop that into product board. Um, we've got, we have a hackathon coming up very soon, so maybe somebody will pick that up and just suddenly you'll see it in the, in the product and a couple of months. Oh, that'd be really cool. Yeah. Looking forward to it. Nice. All right. Do we do we want to open it up for questions? Okay. Hey, everyone. I'm pragnya. I'm part of the product marketing team here at Amplitude. And we have a couple of questions, but if you guys have more for summer and Alexandra, please drop it in in the Q&A box here. So the one question that we've gotten is how. Alexandra, how do you think about real time data versus batch data in how you're thinking about your analytics needs in general within Amplitude and other parts of the business? Yeah. So there are definitely. Use cases for real time. I do find it very, very helpful for sends checks and also for just like calming nerves, especially if, for example, you've just started an experiment and within the first 5 to 10 minutes you just want to see that like the users are being allocated, assigned and then exposed to your experiment. Or maybe you just release a new feature and you want to make sure that you haven't got the error feeds, just like exploding, etcetera. But when it comes to that cultural shift, I think there's also that thing of you don't need to know everything all the time, and especially when you get so much data coming into Amplitude. I don't generally use too much of the real time just because then you just be constantly checking like 500 charts a day, like all day and you wouldn't kind of progress too much. So definitely the side of things. It's nice, those sense checks, that's really great. But then when you're trying to look at things like trends and I come from a forecasting background, so I do like to see what's happened and then be able to extrapolate it out to see how the performance should be going. That's when you start to need to open it up to generally, I'd say daily is the is the lowest that I would go when I'm drilling down, when I'm doing actual analysis. Okay, great. And then somewhere we have one question for you in understanding how does Amplitude think about evolving the data functionality to make it relevant for the data teams? So how are you thinking about that overall? Um. You know, honestly, like one of the, one of the things that we've been focusing on a lot lately is how do we support a wider spectrum of, of the maturity curve. Um, you know, as Alexander was saying, like, we've got great tools in data, right? For planning your taxonomy, managing your schema. We've got our Arcserve capability which allows you to monitor the health of your data as it's coming in. Um, but there's a number of folks that for whatever reason, either they just haven't gotten there yet from a, a need or culture perspective or just don't have the resources to do that, that upfront planning. We're we're really digging into how can we create more of a responsive governance environment. Um, how can we just let people, you know, if they have to turn the floodgates of data on and then we observe that data and give them recommendations that allow them to manage and govern with just a couple of clicks. So that's an area we're digging into, into a lot. So. Meets my needs like I'm lazy, which I guess makes me a pretty good product manager. The shortest distance between two points is a straight line. Um, so. Yeah. So. I'm looking forward to some of the stuff that we're going to be rolling out in the in the next couple of quarters. Trait. So those are all the questions that we have today from the group. Thank you so much, both of you, for your time. This was a really fun conversation to listen to and really appreciate you and our audience's time this morning. Thank you. Thank you. Brilliant. Thank you, everyone. Have a good evening, Alex. Cheers. You too. Bye.
