From AI Spend to AI Accountability

Julia Nimchinski:

Okay, I’m very excited to present our CEO Roundtable. Please give a very warm welcome to our HSC regular, Erik Charles, Head of Revenue at Variabl and former Chief Evangelist at Xactly. Erik, super excited to have you back. How’s your summer?

Erik Charles:

Summer’s going well! It’s a little bit of rain here recently, which we desperately needed, and I’ve been running around the country quite a bit, so it’s good to be, though, back here upon screen, believe it or not.

Julia Nimchinski:

Well, welcome back. Awesome. Thank you. Take it away.

Erik Charles:

All right, thank you very much. We have one hour here as part of the Agentic Harness. I’ve got a great time, you know, luckily, the folks at HSC have set up a fantastic panel. I’m gonna do a fast introduction of our panelists so we can get straight into the questions, what most people want. First, Amanda, CEO and founder of 1mind. I first met Amanda at South by Southwest years ago, when she launched… during the launching period of 6sense. I was a user of that at Exactly, and I have followed what she has done at 1mind, raising money from Battery Ventures.

The AI agent named Mindy, which starts handling everything, which I have to admit, drives me insane as somebody who works in sales compensation. I feel like you keep on trying to get rid of the reps that I try to handle that pay for, but that’s okay. It’s a great tool. Adit is the co-founder and CEO of Lena AI. He’s one of these folks that skipped the corporate detour and went straight into being a founder after coming out of IIT Delhi in 2015. I’ve actually done some guest lectures at that university. Has spent, you know, turning a chatbot into an enterprise platform.

And, you know, he’s got, like, these tiny little client, you know, customers using his system, like Coca-Cola, MongoDB, and those folks over at McKinsey. So, we really have to thank him for taking some time out. Jayla, who’s the CEO and co-founder of Mutiny, ran product marketing at VMware, helped grow Gusto’s revenue, I’m laughing, I have Gusto my phone right now for… we use it at Variable. But most importantly, or more interesting to me, she shut down a massive, you know, 8-figure ARR to rebuild a company as AI native. And I’m really gonna dive into that more here in a little bit.

And she’s now got, once again, agents that are doing 80% of the work of a reps week that’s not selling, which is greatly appreciated. She’s got Snowflake, Uber, and Ripley using her stuff. And last but not least, Amos, just in the order that I threw it out on the thing, co-founder and CEO of Swan AI. He’s built a couple companies before this, had some successful exits, now running probably the most public experiment in the room. on whether a company needs employees at all. Ai go-to-market engineer that can do all of the work behind a revenue team, 200 customers right now, with 3 founders, and 0 employees.

So this is fun! I mean, this is all the stuff that everybody’s talking about. What’s AI gonna be doing? Where’s it going? But this is… we’re not just… we’ll save the last 5 minutes for, you know, prognostication and pontificating, but right now, let’s go into this business model reset we’re all seeing. So, jaleh, and if I’m mispronouncing that, I apologize. I forgot to get, you know, pronunciation guides from y’all when I was putting this together. Last November, you shut things down. and restarted with a tiny team, and, you know, every CEO watching has been told to add AI to what they already sell.

You looked at the same math, decided the old model could not be retrofitted, so what’d you actually see?

Jaleh Rezaei:

It’s pronounced jaleh, so very close.

Erik Charles:

And I apologize, thank you.

Jaleh Rezaei:

No, no worries at all. I’m very, very used to it. So yes, we had a very spicy reset in November of this year. it can be a whole podcast, so I’ll give you the TLDR version, which is that, ultimately, what we learned after a lot of trial and error, is something that is probably now very obvious to everybody, which is that SaaS is a workflow tool, and AI Is automation technology, which then eliminates a large portion of workflows. And so… when we were trying to cram AI into an existing workflow tool, it just limited the level of automation that we could do for our customers.

And, you know, to give you a little bit of context. our big thing is providing a really amazing buying experience, right? And so we wanted every rep to be able to provide that for all of their customers, but in a fraction of the time that they’re spending today providing a crappy one. That requires tremendous automation, and not something that you can try to shim into an existing, existing SaaS workflow. So, we started building that, from scratch. We had to focus quite a lot on brand new concepts that weren’t really a thing in the previous products.

So, like, for example. quality. How can we replicate the company’s brand? How do we mimic a rep’s voice? what does good look like when we’re generating all sorts of things on behalf of the rep, given the customer context on the fly, and making sure that we really get that right. That’s a lot of evals, a lot of training, things that, you know, doesn’t really exist in the non-AI world. And then, I would say similarly for the rev ops and the management layer, a whole bunch of governance and usage and budget management. that are also not really a thing in the SaaS world.

So, we kind of looked at this problem and said the most important things that we need to build, are completely separate for us. It was a… it was a stressful couple of months there when we… when we did that reset, saying goodbye to, like, all of the traction that we had built. But we released the new product in… in April, and already 3,000 people are using it. You know, you mentioned some… there’s some really big brands that are… that are using it, so it’s been, you know, I do recommend that type of focus if you If you see a really big opportunity with AI for your particular industry.

Erik Charles:

I mean, I think it’s an interesting question, because, you know, we can see lots of the legacy SaaS vendors are out there, and I’ll further admit, bolting agents on top of existing platforms, and they’re certainly seeing benefits, there are things you can get done with those agents. But… So, what’s the cost of a… how do you balance, like, I’m gonna add agents, I’m gonna do a retrofit, versus, nope, blow it up, start over, rebuild it from scratch, based on top of the AI? How do you… how do you do that cost modeling if you’re running a company?

Jaleh Rezaei:

Yeah.

Amanda Kahlow:

I mean, did you churn the customers, and did you, like, say. good night to the customers, or did you migrate them over? That’s my… I mean, sorry, I don’t mean to jump in with.

Erik Charles:

No, no, no, no, no, you know what, Amanda, it’s gonna come to you.

Amanda Kahlow:

your size, what you’re doing, sorry.

Erik Charles:

Yeah, not at all.

Jaleh Rezaei:

No worries. No worries. Yes, we… we… we turned the customers. So we had, you know, we had, like, about a 3-week period where we talked to our customers, told them what the new thing was that was gonna come out in a few months. But ultimately, yes, we walked away from the revenue, we changed the business model, to be, to be PLG instead of tops down. We changed the pricing model, we basically changed everything. Even the buyer is a little bit different than what we had before. So, historically, we were selling to ABM teams, and now we sell into sales teams.

ABM still uses Mutiny quite a lot, but we… but we go direct to the… to the sales team. I think, Erik, To answer your original question, we… we tried running the scaled SaaS business, so first you have to decide, is this the same thing? Is AI a feature of the product, or is it a separate thing? And I think if you really want to realize the big opportunity, it’s got to be a separate thing, because you’re competing with products that are, you know, unhindered, and they’re building the greatest possible experience that they can for the customer, so I have a hard time seeing how some hybrid can be the best product.

And so once you decide, okay, it’s a separate thing, then you’re trying to run a scaled business alongside what has to start as pre-PMF. And I just think these are two different modes of operation in my scaled business. It’s all about delegation, and and there’s structures, and quarterly roadmaps, and, you know, all that stuff. And in the pre-PMF, it’s benevolent dictatorship. Founder holds all the context in their head, making decisions as fast as possible. the roadmap changes 5 times a day, and the only thing you notice is that progress is happening somehow a lot faster.

Like, every month, you’re much closer to the output. And so, those are just two different modes, and I… we tried a period where we did both, and then in November is when we made the tougher call of, we think we can just go a lot faster, be more cash efficient, all of those things if we’re just focused on this thing.

Erik Charles:

Yeah, and Amanda, you did the same thing. You started a new company, you went AI native. You know, and so, you know, originally what I was thinking is, is retrofit ever the right answer? But, you know, is every incumbent SaaS company now just a simple legacy cost structure, just waiting till they blow it up and rebuild it?

Amanda Kahlow:

You know, I mean, it’s interesting. I mean, you look at, like, what Salesforce is doing, which is quite interesting with Claude Force, right? Putting that on top of the system of record. I think, you know, historically, we were systems of record and systems of engagement, and we’ve moved to, you know, as you were saying, systems of outcome and systems of workflow, right? We’re looking for what can we do at the end of the day to drive… to move the needle forward. At 1mind, just, we’re building superhumans, we basically replace the concept of an AE, an SCR, a sales engineer, a technical CSM.

So we think about AI being the seller, not supporting the seller. I think the historical world was everything was built around how do we, especially in go-to-market, how do we enable sellers? How do we find pipeline for them? How do we prep them? How do we get them trained? How do we give them coaching? How do we get them to get the next best action, spend more time in the field? And I think my thesis is, actually, AI can do more. It can actually do the selling. It can do the work, and it can have the conversation.

And so I think the world is moving towards, you know, how do we really think about outcomes versus thinking about processes, and then not process around people, but process around outcomes. So it’s… I mean, I think there’s some good, like, I mean, Clay is not an AI-native company, and they’re doing great right now, right? So, you know, like, there are some go-to-market technologies that are… I think, like, 6sense has the opportunity to shift and to go into that world. And I think, you know, obviously, I’m still bullish on my old company. I’m not there anymore, but I think they have a moat as far as the data goes.

So I think there’s still options, but you have to make bold moves. And so, really impressed with what you did, by the way.

Erik Charles:

Those data modes, we might have to get into this later, because I think that’s going to be important. Thank you, y’all, and we’re going to keep rolling pretty quick, because I want to make sure I’ve got all the voices. I have to say, at Variable, you know, this is my third time in the incentive compensation software space. I was at Calidus. back when it was the first dedicated on-premise solution, where a sales engineer said, how big of a Dell box did you need in your in-house rack? And then I was at Exactly, which was one of the first, if not the first, SaaS solution, and now I’m at Variable, which they rebuilt the whole thing, starting with AI, and it’s been amazing to see how quickly it moves.

But, of course, building the software is one thing, that would get me to the pricing game. And so, Aditem, this is where I wanted to jump to you, because… I mean, how dare you go away from, you know, Don Benioff’s, I will charge you this much per seat per month, billed, invoiced quarterly, for the rest of your life, and move to an actual platform fee with consumption. I, I just… tell me… you’ve got to have all sorts… that must be fun, like, even doing, like, you know, revenue forecasting. Your board must be going, how much we gotta make?

Well, depends on how much everybody uses, really. So, walk us through this. I mean, you’ve got variable revenues still under 10% of ARR, which is interesting that you actually went public with that, so I’m not making that number up. What’s going on here? Where are we in this consumption world?

Adit Jain:

Absolutely. So, jaleh, by the way, I’m sorry if I’m butchering the pronunciation of your name. We are YC Batchmates, so you went to somewhere 18, we were at somewhere 18 as well, so, hey.

Jaleh Rezaei:

Oh, amazing.

Adit Jain:

Yeah, we should probably catch up separately. So, and we’ve got… we’ve gone through a similar journey, so just to give you some context, We started in 2018 with the idea of building, you know, a Jarvis for employees. So, IT HR chatbots to reduce tickets. That’s how the company started in 2018. We did really well in the pandemic because ITHR finance teams were not in offices anymore, so people had all of these questions and all of these problems, and they needed… so that’s why, like, the business took off in COVID. But then LLMs happened in 22, 23, and we were like, oh, damn, all of our product and technology is basically useless, right?

So, we… like, the big difference is, you know, we’re not a SOAR, we’re not a system of record. We were always a conversational interface on top of multiple systems of records. But the technology underneath us just shifted completely with people accessing ChatGPT, and we realized like, all of the experience that our end users have is suboptimal to what they’re having in the chat GPT era, so we need to change, and we need to change fast. So we decided to kind of get rid of all of our old-school technology, like, which is built-in NER models, so people who are close to machine learning will understand that.

So we had to kind of rip all of it out. in 2023 and 2024, while our customers were still using it. So, while we had a few customers, and they were still live, they were still using the ITHR chatbots, but we had to kind of rip all of that out and become fully Agentic in 2024 itself. So, it was a fun journey, it was, like, it was not… It was not easy because, you know, we were not building from the scratch. And we had to make sure that our customers’ configurations and integrations with SAP, Workday, Salesforce, ServiceNow, all get carried over to Lena seamlessly, right?

So it was not that easy. But, back to your question, Erik, what happened is a lot of our customers throughout 2024, 2025, and this year started using the platform to actually build a lot of AI colleagues. So that’s what we call agents in Lena. So, they started to build a lot of AI colleagues across back office business processes. So, think IT, HR, finance, think about onboarding, offboarding, accounts payable, accounts receivable, all processes which are very manual, need a lot of humans to kind of do them, and very repetitive in nature. So, our customers started to use our platform to actually build all of these use cases out. while we had actually licensed them the technology to use it for HR and IT assistance, to reduce tickets coming into ServiceNow and Jira and all of that.

But they were starting to use it not only for that, but for so many other things, and we were like, oh, this is… A, this is amazing, but, you know, the token consumption bill was killing us, because our customers were just using it and, you know, going all over the place with this. And… at that point, we had to, Erik, decide to kind of go consumption, because it was so hard, like, the use case could be anything. how would you… like, the use case could be onboarding, it could be offboarding, it could be… it could be any kind of use case, accounts payable, in the back office specifically, and then we could not, like, we don’t… we didn’t want them to call us and ask us the price of that use case, right?

So the best way for us, and things that CIOs and CEOs also told us, and CFOs as well, that, hey, I don’t want to be logged in into large contracts, and I want to, you know, have predictability in my pricing, and I need to be able to use… I don’t want to keep calling you every time I want to use a new use case, right? So, based on that, we decided to just, like, simplify, and, you know, I looked at pricing. I remember looking at pricing of all large platforms, including Salesforce, ServiceNow, SAP, Workday, all of them, right?

And they started to come up with, you know, some sort of usage-based pricing, but Everybody was trying to build their own like, units of measurement of consumption. I think that’s a very bad idea, because that kind of increases confusion in the minds of buyers, so we just went back to tokens. And we just priced it on platform and consumption, 100%. So, after that, that has happened, you know, we’ve seen a lot of customers, you know, like, understand and just choose us because of ease and transparency of pricing as well, Erik. So, yeah, it’s been hard to kind of… tell you that how much will a customer use next month or next quarter, that’s a little hard, but we are kind of building that muscle.

And my sense is, if you have a wide array of use cases, you will have to build that muscle any which way. So better start early and start right now instead of later, because, that… that… whether you call it token consumption, you call it credit consumption, whatever you call it, like, there’s some unit of consumption, and you will have to build understand how… customer usage patterns grow over time, and better start early than later. So that’s been our journey. It’s been really super exciting for us, you know, over the last 12 odd months.

Erik Charles:

It’s interesting, because I’m looking at this right now with seats for our engine. Mainly, it’s around people that have, like, you know, third-party distributors or independent agents of, like, do I really charge for a seat every month when a lot of people don’t sell anything that month, or… and also highly variable sizes of sales forces, like, say, in luxury retail, where they hire a bunch. So we’re… I’m already putting… variable seat licenses in front of my customers, and even trying to examine, even going further, I laughingly said to one, what if I just, charge you a percent of the revenue? you know, the percent… or the percent of the commissions, so that you know the charge to me goes up or down by how much you pay out in commissions.

But jaleh, you went… you also were on seats. And then you went to an AI credit model. So, what was that transition like, and how’d you figure out how much to charge?

Jaleh Rezaei:

Totally. So, I don’t think we have the right answer yet, but I’ll share our journey and learnings along the way for what it’s worth. So, we… we initially released a seat-based model with credit limits per seat, and then you could opt into a credit pool that people could consume and draw down on. We changed that to a credit-only model, and credits don’t translate directly to tokens, but it’s related. And the upside of that shift for us was that we made it so that it was free, effectively, to add somebody else into Mutiny, and that had a big impact because, you know, we went overnight from a couple of users at different accounts to dozens, hundreds of users, and that was really good, because now we have more people that are using the product.

But the downside that I still think we need to solve is that it has made procurement more stressful, and the customer has to now predict usage, they don’t know how to predict usage, and so it’s created this challenging experience at renewals, or when someone, you know, if it’s a new customer, they almost always go for the minimum, because they’re like, well, what’s your minimum? I’ll get that And then we’ll… we’ll learn about the usage. So you mentioned the seat, you know, having software that people aren’t using the seats. I’ll give you an example. So, we are right now displacing, HiSpot, which is a legacy sales enablement tool at an enterprise company.

And they wanted to try Muni, because a bunch of their reps were using it, and they were like, hey, we should use this instead. Their High Spot usage was very low for the seats, and so they ran a pilot, 90% of people were using it. everything’s great, right? But then it gets to procurement, and it’s like, okay, well, I paid this… I have this many reps, I paid this amount per rep, and that’s very predictable. Now, with Mutiny, yeah, I pay nothing if they’re not using it, but they also seem to really like it, and they’re going to use it a lot, and… you know, how do I manage that?

And so the way we’re handling that right now is through lots of good modeling and trying to translate the work into credits. But what I would like to get to is much closer to outcomes, like Amanda was mentioning. I think it’s really hard to get to revenue, at least from sort of where we’re sitting today, but getting to outcomes that the customer can still understand, measure, and verify is definitely feasible. So, like, for example, for us, that might be follow-ups. Today, it takes you 1 hour to do proper follow-ups for an enterprise account, which is where we primarily focus, and that’s 100 bucks.

In terms of rep salary, we can do that for 10 bucks, and in, you know, 10 minutes. That’s the place where we are now exploring and trying to move to.

Erik Charles:

Yeah, Aditi… Comment?

Adit Jain:

Yeah, so, I have a question, jaleh, and So that’s coming back to the point I made around, you know, credit models and the complexity in how do you calculate them. What piece… what is a unit that will, you know, draw down how many credits? And that becomes a big question always. So, why don’t you go just token consumption? Just a question that I wondered when we decided to go just token consumption, but just want to hear your thoughts.

Amanda Kahlow:

I can share why we’re not doing it, but go ahead.

Jaleh Rezaei:

No, go ahead, go ahead.

Adit Jain:

Gross margins? Is it gross margins, Amanda?

Amanda Kahlow:

No, I mean, so… because that’s not what… that’s not what I’m selling to my customers. I’m selling an outcome, and I’m selling a seller, and I’m selling an experience, so the way we do it, we… so basically, if it’s a sales engineer, it needs to be on calls, it needs to be answering questions, it’s… So we… we basically go back to the seat model for the sales engineer use case. how many people are using the Superhuman, how many calls, we estimate how many calls, and then I do something called a fair use cap. So, I basically say, you think you’re gonna have 2,000 a year, or a month, or whatever it is?

Great, I’m going to give you four. I’m going to give you 6,000. So I’m gonna give you way more than you think, because I don’t want to limit usage. The worst thing I can do in year one is for them to say, oh, shit, I don’t want to use it, so I’m not going to bring it on to calls, or I’m not gonna put… I’m going to put it on just one page on the website versus everywhere on the website, right? So we want to… or we also live in product. I don’t want it to just be in obscure places within the product.

I want it to be pervasive. So in year one, if I lose money on my margins, I’m fine if I have a happy customer, because now I have a lifetime customer. And I’m showing outcomes and value, and so I need to make sure my pricing aligns to the business value that I’m selling, not to how I am charged based on the foundation models.

Like, and also, the other piece of it, too, is, like. we as, I think as vendors, need to be thinking about how we are consuming those tokens, and the models, and whether we’re using open source, or whether we’re using, like, you know, the core foundation models, we have fallback models, like, for us, in our world, like, we use OpenAI, but we fall back to Gemini, and we also have open source, and then we have multiple voice models, and when one is down, I mean, if you all noticed that in the world of OpenAI, when they release something new, everything slows down.

And so, in my world, I’m speaking, I can’t slow down. Like, if I slow down, like, you’re like, is the thing fucking alive? You know, so we need to, the minute we see that there is a, like, it degrades, like, we need to fall back and go to the next best model, which might be more expensive, because I’m always managing cost. latency, and and response times. So all those things, three things are, like, we’re playing Jenga at all, like, the same time, so I don’t want to put that dependency back on my customer, because, like, yeah, sure, I could, like, charge a bunch more and make this super fast, but that’s not the goal.

The goal is to make it really good. charge less and be super fast, right? So, like, we want to have all those things together, and that’s my responsibility. So, I really think it has to be aligned to the business that we’re selling to them, which is, I’m selling that this superhuman can sell, and can sell just as good as a lot of your sellers.

Erik Charles:

So, alright, are you… so this kind of ties into a question that came in from the audience. Does this mean you’ve got a digital 1099? Are they on a variable comp plan? I mean, for an outcome-based? I mean, do you… Do you take a piece if they actually successfully close business for your client, or is it just your turn?

Amanda Kahlow:

customer doing that. I have one customer that they wanted that, right? They’re like, alright, we just… so we’re gonna pay you based on the success. It’s too early to tell whether it’s working or not. The hard thing about it is every customer is different. So I sell to companies that have PLG models, and that works on the website. We drive free trials, we book meetings, like, then I have SLG, which is on a call, like, a sales-led model, so everybody’s outcome is different based on their business, so it’s really hard for me to go to, like, this outcome world in, like, a traditional, like, okay, if it’s gonna be, like, a basic chatbot, like a qualified that books meetings.

Which is a… you know, there are people who use it for that purpose. sure, I can… I can do it based on an outcome of booking a meeting and booking a quality meeting, but I want to do so much more. I want to go to the close, I want to give the live demo, I want to be the solutions engineer, I want to build content for you, I want to build your business case. All of those things, like, take up a lot more tokens and a lot more time, and it’s less, like, there’s no outcome to that.

It’s actually just moving it along in the sales funnel. So you could do it by stage progression. Like, we do look at one of the things we measure is the influence on, do deal cycles shorten? Does ACV go up? Right? Like, do these things, when they engage with a superhuman, what happens to the net of your business? Not just the top of funnel. Everybody is focused on top of funnel, and I just think in today’s world, that’s so wrong. Like, we should really be thinking about what is the big picture of the impact that these experiences have on the buyer, and ultimately. it’s buyer… buyer delight.

It’s not even outcomes of revenue. So if I’m making your buyers happy, you’re gonna win as a business. And so that is my North Star, is to make a better buying experience, not to create a better sales motion, which is what happens as a result.

Erik Charles:

Right. I think this is gonna get interesting. I think outcome-based. pricing sounds great until you get down into the great, how do we objectively measure the outcome, and how much is it the… fault or the benefit of the AI versus the product you’re trying to actually move. You know, it’s kind of, you know… holding me if you have a poor product. And actually, I think this kind of fits into another follow-on question, errors. I mean, AIs hallucinate, AIs make stuff up, they’ve gotten a lot better. proper prompts can make… have such an impact. I spent this weekend with two members of faculty discussing AI and emotion, how emotional can an AI be to actually manage emotions, to go down that path of, does it have fear?

Had a hilarious conversation with Claude over a bottle of wine, and these two members of faculty as we’re trying to put together a symposium for next year. So. With the models working differently. An error impact, you know, in, like, creating the index for my book is minor. an error impact in some of the models we’re running can be catastrophic. So, how do you do those use cases, Adit? How do you manage that? And ever.

Adit Jain:

No, great question.

Erik Charles:

And afterwards, too, so… yeah.

Adit Jain:

Absolutely, you know, great question. I think, this is something we’ve been thinking about deeply, because our, like, the type of business processes customers automate using Lena could be, like, really critical processes, like payroll. Right? Like, payroll processing is a… is a big use case that our customers automate on LENA. Then, of course, accounts payable. So you could just pay Erik a million dollars instead of $10,000, right? Like, and so… so, like, we’re managing real cash out and in, accounts payable, accounts receivable, payroll, etc, right? Even, you know, processing all of your reimbursements and travel requests, all of that, right?

Expenses. So, there’s real cash involved. So, in those cases, the way we look at it, Erik, is, we’re running… we are grounding the model in your company’s business process, so let’s say we have Coca-Cola as a customer, Estel Order as a customer, or whatever as a customer, so… your… process for accounts payables is these 10 steps with these 8 approvals, and then somebody else’s is 3 steps with 18 approvals. So whatever your business process is, we basically put that in, so you can upload a document or just write it in English, simple language. It’s called an AOP, Agent Operating Protocol, which is like an SOP, but for AI colleagues, right, or agents.

So, what happens then? The model is responsible, or model is grounded into those extra set of AOP instructions, Agent Operating Protocol instructions, and then we have a checker model as well. So, while this main model is following this business process. There’s also a checker model that will keep looking at the reasoning tokens. and the AOP input it into this, and check whether the first model, which is the main model, is following the instructions or not. And if there is deviance, you know, it just shuts down and it goes to a human for checking, right? So, it’s not like you’re… let’s say you’ve done a million runs of an accounts payable process, so the ones where it is deviating is the ones which will escalate to a human manager of the AI colleague, the rest will continue.

So, only that one will get stopped or paused and go to the human manager of that business process. So, that’s basically how we’re kind of doing this, and it works really well, because if you look at the numbers, and we’ve run those numbers ourselves on many benchmarks. A single model typically hallucinates 2-3%, two models together will hallucinate, or at the same point, or at the same reasoning step, or at the same step. the probability of them hallucinating at the same time is around 0.04%, if you just multiply them together. But then, the interesting thing is they have to also hallucinate the same thing in order for it to pass through.

So the probability of that happening is, like, way, way lower than 0.04% as well, almost next to zero, because the only way it passes through is they hallucinate the same thing, if you have two models, from a maker-checker perspective. Works really well at scale, we’ve tested it on millions of Runs… business process runs and automation runs, and it works really well.

Erik Charles:

Okay.

Amanda Kahlow:

I mean, like, my perspective on hallucination, like, in go-to-market, right, you can’t hallucinate in biotech, you can’t hallucinate in financial services, right? So that, like, there’s… room for error is very small, because you’ve got lives, and you have real money on… on the line. in go-to-market, like what we’re doing, hallucination, if it’s hallucinating 2%, like, I always say the benchmark is a human. How often do humans hallucinate? Probably. 40%, right? So you kind of think of, like, the difference of a human versus an AI in this situation, and so with tight evals and guardrails, you know, as Adit was saying, like. it’s… it’s a non-issue.

I mean, people worry about it because you’ve heard, like, the bad actors of, like, the early days of people using AI and not putting it on good guardrails and not knowing how to do it. But I think we’re way past that now, and I think that’s a fear that for me, we don’t… I don’t have any of our customers… I’ve never had a customer come back to me, ever, in our two years in market and say, our AI is hallucinated in any way that has negatively hurt the business at all. So, like, there’s… the only thing is that they’ve given us content that’s out of date.

Well, you know, I can cite where it came from, here’s where it came from, that’s what it said, it’s… it’s citing that content, which is an issue, like, which is a really big challenge, and it’s hard for everyone, not just, like, our are superhumans, but everyone in AI is like, how do we manage? Like, one thing we’re constantly grappling with and building towards. is how do we keep information up-to-date as the pace of AI is moving so fast, and, like, we’re releasing features so fast? How do we get that information into our sellers’ hands, and into our superhumans’ brains, and in an accurate and up-to-date way?

What is the most up-to-date content? it’s a really tough, problem to solve that I think we’re all trying to figure out, and I don’t think anyone has the answer to right now. Yep.

Erik Charles:

Let’s go into the next one, which is good ol’ headcount, and AI is taking all the jobs, and, you know, people right out of university can’t find jobs because entry-level professional jobs are run by AI and the like. And Amanda, I’m going to bring it back to you, because, you know, it’s always fun to say, because, you know, you scare me whenever you say we don’t… you can replace the AEs and replace a lot of the salespeople, because those are all my seats that I’m trying to sell. how close are we? You know, what role is going first in the organizations?

I’d love to hear everybody else. I mean, honestly, tell me, what jobs are you, quote, taking away, but at the same time, I’ll give you the… what jobs are you creating?

Amanda Kahlow:

Yeah, yeah, so I think there’s two ways to think about this. One, there are, like, on both ends of the spectrum. The jobs that are going away are on both ends of the spectrum, like the really highly skilled jobs and the entry-level jobs that you mentioned. So some entry-level jobs, like the BDR and SDR job, obviously those are going away and getting replaced by AI. And on the other side of the spectrum, like, what we’re looking… what we’re seeing with our customers is there is, like, a sales engineer or a solutions engineer. We’re not doing the whole job, because there’s, like, building custom demos, there’s responding to the RFP, which eventually, there’s nothing to say that AI can’t do that.

We’re just not doing it yet. So I think there is a world where actually we will take that as well. But where those people fall down is… imagine, you know, when a company sells to multiple different verticals and industries, and they have a complex product, no human can take in that information and really know the buyer, everything, not just the company of the buyer or the products that they sell, but the individual that they’re selling to, plus know everything about their own products and services. So, like, think of, like. Cisco, for example, has, like, incredibly complex products, or, like, Rockwell Automation.

Like, you think of these companies that have these massive product SKUs that no human can take that complexity. We have limitations. And so I think in that area, AI does extraordinarily better than a human, because we can’t maintain… like, I get on, like, a very simple example. I get on so many sales calls and help, like, close the deal, because I have a lot of relationships, and I’ll jump into a call. My sales team will send me the prep, because when we haven’t replaced the AE yet, so they’ll send me the prep doc. it’ll be 8 pages.

I’m lucky if I retain 5%. I’m like, alright, what’s the business model? What are we selling? What product are we selling? Got it. Okay, and that’s what I can get on, because I have 15 calls in that day, and I just don’t have the time, the capacity, or the recall. I’m not smart enough. to remember what I just… even if I read it. I don’t remember it, to actually bring it in at the right moment at the right time. AI can’t. And so that’s what’s so beautiful. You can prep the AI, you can give it massive amounts of information, and it can use that information skillfully and artfully to, like, move and progress the deal forward.

So I think that’s how we have to think about it. So I think the place where AI is really coming in is a lot where there is no business model to support a human. So another, I’ll give you one more quick example. As we think of, AI, like, on both ends of the spectrum. We think of, you know, there’s obviously the PLG motion, where you can help, you know, like, everybody who has a PLG, like, you can get, you know, churn, get those customers in and out really quick, and then there’s sales-led. But there’s the customers, the companies that are selling commercial and SMB.

Those companies that are selling commercial and SMB, you still need a human, and it’s really hard to staff that human into that job, especially if you have a complex product. So if you have a down-market product that’s a low ACB, that has high complexity. Humans struggle. So you struggle to find a business model that works. I think that is a beautiful spot for an AI. So I think of it as, like, where is no business model for the human? There are tons of jobs right now where the AI is doing that. So, while we’re not replacing the strategic enterprise sales-led motion AE, we are selling commercial deals for our customers as a commercial AE, right?

So a commercial down-market SMB AE, and then we’re also doing on the PLG side as well. So, kind of thinking, like, we can throw content at them, or we can have a conversation. Everybody wants to talk, like, that’s how we buy, that’s how our brains work. We want to ask questions, we want to be… feel heard and feel solutions, and I think… so we think of it as, like, where there’s no business model, both ends of the spectrum, but ultimately. there’s nothing stopping us from going there, it’s just the complexity of building the product to get there.

The product just isn’t there yet.

Jaleh Rezaei:

I’ll second, the piece of that on where there’s no business model, that’s a great place to put AI. So, when… I was running sales marketing teams at Gusto, I would have to A-B test the human touchpoints that we could afford. So, okay, if we give the customer a call, right, when they sign up, what is the lifting conversion rate? What is the cost of the human minutes spent? okay, we are allowed to insert this playbook, right? I would have loved something like OneMine, because that would have helped us tremendously if we could just answer people’s questions, without having to just only always find the ROI-positive one.

I think that’s a great entry point. And then from there, you see, okay, where else can this go? This is much better than we thought. And you expand from there. I think at the end that we serve, It’s… The models are getting much better at getting the correct answer, but what they are missing that I don’t really see, getting fixed anytime soon is intuition. So, in enterprise, there’s so much… oh, I sat in a room, and I can tell that that guy is the decision maker, and I can tell that everybody listens to this person, and even though this person’s title is really high, they don’t actually… they’re not, you know, they’re a detractor, or whatever.

And so there’s quite a bit of judgment, so even though we have gotten better and better at producing materials that the rep has few edits on. The way that they weave in the stakeholders and influence, like, all of that stuff is still, manual and uses, you know, their… their… just their intuition, that gut feeling of, how should I structure this differently, etc. I don’t know, Amanda or Adita, if you’ve seen different, approaches to things that require that kind of intuition.

Amanda Kahlow:

I think AI has great intuition. Like, we are… we have multi-speaker diarization, and it understands who it needs to be talking to, and it’s pretty wild, and to be able to pull that in into the moment. You were gonna say something, sorry.

Adit Jain:

Oh, I saw you smiling, and I knew you did not agree to what jaleh said, but anyway, so… so my… my… so we’re not in the go-to-market space, we’re more on the back office automation, so we’re definitely seeing that problem of entry-level jobs going away, Erik, so… So, if you think about it, like, from in the back office, you know, you… HR, IT, finance, you get a degree in finance, you have an entry-level job, right? Today. those jobs are going away. Like, record to report, there’s a business process in finance, right? Like, that used to require a lot of, like, number crunching, Excel. it’s almost gone at this point, like, AI can do all of it, right?

Of course, there are complex businesses, which have, like, 100 entities, 2,000 product SKUs, that’ll probably require humans still, but… those jobs are gone, right? Like, if Lena is implemented in a large customer, enterprise customer, we’ll get rid of… like, 70% of tickets coming into IT and HR in 6 months, like, we guarantee those things, right? Like, in contrast, like, it’s on us, like, we will make sure it happens, right? And that’s also part of our outcome-based, you know, pricing approach, but… I do worry on what happens to, like, people who are just coming out of college, right?

And, The good news is that there are new types of job roles that are also getting created, Erik. So, for example, one thing we do is when we go into a customer, we say, hey, whatever business process in the back office you want to automate, let’s say you want to get rid of the IT and the HR help desk, right? So go find the best employees in your L0, L1, L2 help desks. And then we will give them a course, like we have an LMS, so they’ll go through a training course, certification. You should typically give them a pay bump, around 30-40% pay bump. and then make them the human manager of that AI colleague, right?

So now it becomes their job. It does two things. Number one, it motivates the person to share as much as they can about the process, because I think Amanda mentioned this, that, hey, knowledge is not that great all the time, and business processes, which we put in AOPs, are not in documents anywhere. It is in the mind of the person who’s been recruiting for 20 years. who’s onboarding, or who’s doing accounts payable for 15 years, right? It’s in their minds. It’s not written down. Their business process is not written down anywhere.

So if you do this, what it does is it encourages the person to open up. and share more about their, about that business process, because they’re making more money, and then you can basically comp them on the outcome, which is, hey, if you get to 50% automation, if you get to 70% automation, if you get to 80% automation, this is how much bonus you can get, right? And then they’re fully motivated, and their goals are aligned with yours as an organization. So, you can basically say that for every 10 people replaced in any of these business processes, you give back one or two jobs with new types of things, like, you know, as I mentioned, human manager for the AI colleague, or in the center of excellence that you create for AI colleagues in large enterprises.

I’m talking about really large enterprises right now. So, but yeah, like, I do worry on what happens to some of these degrees, and maybe we look at a future where they have to stop existing. Some of these… these… even degrees and colleges have to stop existing, and you have to look at some other type of skills and training for people. Because, hey, you know, I come from India, my… if it was not clear, and a bunch of these BPO jobs are back… back home in India, right? Like, back in India, so I do think about it, and it’s… it’s interesting, I think. education and degrees and the concept of… and their concept kind of has to change.

Some of these things are not relevant anymore.

Erik Charles:

I’m seeing it with system integrators. The GSIs are starting to feel the pain. You know, they would do these massive data integration projects for software installations, and guess what? The AI can do all that. you know, a lot faster in the 9 months to get the data streams clean. Jaleh, did you have something you wanted to add to that? Sorry, didn’t mean to cut you off if I did there.

Jaleh Rezaei:

I know.

Adit Jain:

I’m sorry if I may add one more thing. So, it’s not all bad, though. Like, I do think there are a lot of different opportunities that are opening up with AI, so I think jaleh and Amanda mentioned that there are things which made no ROI sense earlier, but now they do. I think the same is in the back office, too. So, for example, there’s a healthcare customer of ours. Now, as per the federal regulation here in the US, you cannot, like, you have to make sure you have updated information of all your patients. So, if you’re a large hospital.

You have to make sure that every patient that visited you in the last 5 years, or still continues to visit you, you have updated name, address, email address, address, phone number, and you update that once a year. Now, nobody could ever comply with that, because it was too expensive. to call and make sure that somebody picks up, and then you get that information, right? But now with AI, you know, customers have actually built these AI colleagues that are going out and doing this, so how… so it was impossible. They were just paying the fine out if they were caught, and living with that, but now, because it’s become possible, they don’t… they can comply fully.

So there are new opportunities for these SIs and for, you know, all of these people, but I think there needs to be intentional thought put into this, you know, and what happens to Two people just completing college right now.

Erik Charles:

Yeah, I think it’s going to be interesting. I think the labor arbitrage of nearshoring and offshoring is going to be impacted. In a lot of places. The other one, when you were talking about the intuition, it was, if you haven’t seen it, the World Series of Poker is a habit of mine. Sometimes clients, I’ve got a friend of mine who tries to enter… goes into it every now and to see how far he can make it. ESPN is now showing an AI that analyzes the faces of the players and does predictions on who’s bluffing.

Now, that is the first series, and you have enough video of those individuals to do the read, but it is starting to come into play, and, you know, add in a pair of metaglasses or something, or equivalency at the poker table, I’m gonna see those being outlawed of, you know, turn in any digital glasses so you can’t figure that out.

Erik Charles:

Alright, Amanda, you have built an amazing company with 3 employees. We built Variable with a handful of employees to replace the companies that have a thousand. So what’s stopping, and this goes to everybody, I just, I love, because of how tight Amanda’s kept the headcount on. What is stopping…

Amanda Kahlow:

That’s… we’re not 3 employees.

Erik Charles:

or I think that was at one… maybe it was at when you launched, sorry.

Amanda Kahlow:

We have 100 employees.

Erik Charles:

You’re at a honey, sorry.

Amanda Kahlow:

That’s what we’re doing.

Erik Charles:

I think I must have pulled that from when you launched. That’s my… that’s, that’s my fault. All good.

Jaleh Rezaei:

I think that was maybe Amos on the call, but anymore.

Erik Charles:

Yeah, we lost on Amos, unfortunately, you’re right. Yeah, with Amos for Amos.

Amanda Kahlow:

Amethy.

Erik Charles:

Sorry about that. I had the AM, and then I had lined him out. It’s too bad, I wanted to bug him on this, but that’s alright, I still want to bring it to everybody. It’s getting easier and easier to build things in AI. I… I put… I called into a company recently that is… using Claude to build their own CRM to just get rid of Salesforce, and they’ll have it dedicated to their own platform. you know, and they’ll… I’ll be blunt, they were also saying, yeah, you know, we might look to licensing yours, or we might just… again, use AI to build your functionality.

So, how do you protect your space when you are… doing things that are destroying a space. I mean, how do you convince companies, how do you stay right ahead of that? You know, we’re talking about token, we’re talking about consumption, we’re talking about outcome. when there’s somebody… some CEO says, why don’t we just hire a couple people? And I don’t want to say vibe coding, because it’s been overblown, but you know what I mean. So, who wants to go first on it?

Amanda Kahlow:

Yes. I’ll… I’ll… I mean, I can…

Erik Charles:

Since I misquoted you, glancing at my notes, which makes me, you know, obviously I’m not an AI.

Amanda Kahlow:

Yeah, I mean, so, like. Yes, so you… one is to make it enterprise-ready versus just vibe-coding it and getting it out there, and so we can all talk about, like, the enterprise, like, keeping up to date and making sure that we’re… so it’s just not going to be enterprise-ready, and it’s not always going to be up on the latest, so just keeping up with the latest is one of the basic things. But if I break down, like, our superhumans. I would say to a customer, we’ve been in the build versus buy conversation, and touch wood, over my two years, I’ve only lost once to the… to the build conversation, and… the company that decided to do it, if you knew who the company was, you would understand why they decided to do it.

But the… the basic… yes, you can build a basic chatbot with AI very easily. Question, answer, book meetings. maybe even throw a face on it and throw a voice on it. However, like, when you’re trying to manage, like I was saying before, trying to manage latency and manage accuracy and creating the evals and the guardrails, and then for us, we need to stay out ahead of, like, the next… what are some hard problems to really solve? So I think all of us as AI companies, we really should be thinking about what are those hard problems.

And for us, some of the hard problems we’ve solved are the ability to give the live demo in co-browse. So our Superhuman now, in product, can actually click around on the screen and take over the user screen. Not just giving a live demo and doing the clicking on their own instance, but through a Zoom call, can actually do things for the customer. So that was a very challenging problem that took several, many engineers over 18 months to build, and still is not fully there yet. So we’re almost at the co-browse capability. It is tough.

So, like, those tough problems, or speaker diarization, like, truly knowing, like we were talking about before, who is speaking, what they care about, what their, like, their influence and their sphere of influence, and when you should talk to somebody, and how to talk to somebody differently than somebody else. So, I’m a fast talker from the East Coast, like, and I’m a… IN… what was my INTP? I always forget what the freak my Enneagram or my, Myers-Briggs is.

Erik Charles:

Fire.

Amanda Kahlow:

Not Enneagram. I’m an Enneagram 8, but But anyway, so, like, based on that, I’m gonna say certain things. So, like, the understanding those qualities, and then putting that into a superhuman as well. Really tough to be able to bring all those things together. So, staying ahead of, like, hard problems, and then having, like, fallback models, like I’m saying, like, okay, sure, we can all build it with OpenAI, but you want to put a chatbot that stops talking, because OpenAI just went out with the release of the next model, and you don’t have a fallback system in place, so that’s not enterprise ready.

So we can all build, like, a basic version of Q&A, absolutely go for it, but I’ve got 60 engineers working around the clock to keep an enterprise system stable for my 100-plus customers. You know, and it’s a very, like… that becomes a really easy conversation to have once you get the engineers on the other side of it that are thinking about building this. But it just means that we have to… always be staying ahead of technology, right? Staying ahead of where, like, the, you know, the foundation models are coming out, and, like, their new releases and what they’re capable of.

And we need to be doing more with them and solving difficult things, and that’s what gets me up every morning, is, like, what’s… what’s the difficult thing that we can solve next that’s actually going to drive business value and outcomes?

Adit Jain:

Yeah, so just to quickly add to what Amanda said, I think it is depth. So, depending on where… so, Amanda is building in the GTM space and trying to automate AEs and SEs and technical CSMs. We are building in the back office space, so, hey, like, can we go deeper into what a finance person does, what an IT person does, what an HR person does, and automate the entire business process, right? You can automate, like, 5% of it, but, like, that still needs a human to close, right? So if you… can you automate 80% of it?

And that’s where you certainly start to unlock a lot of magic, because you can really then think of And people, like, enterprise can now think of replacing the human, right? And, because if you were to automate just 5%, then, like. does it… is it real value? Because I can’t really, like, yeah, you can probably have two more coffees during the day, but I can’t really replace that headcount, right? But as those numbers go up and up and up to, like, 60, 70, 80%, and that comes with depth, because you have to then think about exception… exception handling, integration with other applications, data cleanliness, and all of that.

You have to think about so many things. to make sure that it works even those… in those environments and in complex enterprise ecosystems, and still delivers the automation, like, 60-80% automation. So, I think that’s… that’s what is the biggest thing, and build versus buy has been a conversation for every software since the start of software, so… but I would say it’s easier to defend than people make it to be.

Of course, if you’re gonna put in a wrapper on top of something… though, having said that, like, Open Router was literally a wrapper on top of all the models available, but it still did, like, got acquired for $7 billion, but… I, I think if you go for depth in whatever you… whatever you want to automate, if you go deeper, and you really make… you really do that 50, 60, 70% automation, like, it makes no sense to build.

Erik Charles:

Jelly?

Jaleh Rezaei:

I agree with a lot of those points. What I’ll add is. the two things that tend to impact whether you build or not is, number one is, is it an important enough problem, right? If it’s a really important problem, you have to get it right. If it’s not, then who cares how you do it? Just find the cheapest path. But I think the other… the other piece of it is… Is this the place you should invest? That is the highest differentiation and value for your company. And the answer is almost always no, if it is not your core product.

And so that’s the thing that I think we need to come back to. I would say probably starting earlier this year, is when a lot of the build versus buy, decisions started to happen on the… certainly on the go-to-market side for engineering. I think even last year, we were seeing a lot of that. And so I think we’re just in this stage where you can build it on your own, potentially, and so everyone’s going through the, oh, well, let’s be diligent, and let’s make sure that this isn’t something that we can build. But as you go through it, and you look at the complexity of, you know, take something super simple, right, in our workflow, like, we have to process, record the meeting, process the transcript, etc.

There’s models that do that a hundred times cheaper than other models. You would not know that if you are not deep, deep into this problem. And for every one of those, there’s a thousand more of those decisions where a group of people obsessed with this very important problem for your company that are finding the right optimizations, that are trying to give you every single edge so that you don’t have to to know all of the weeds. We do. I think you’re gonna get a much better outcome out of that. I was talking to an enterprise, company who was, you know, he was like, oh, we have 21 engineers building some aspects of what Mutiny is building.

And… And I went through it, I was like, oh, amazing! So do you do this? And he’s like, no. I’m like, do you have analytics? Like, no. I’m like, do you have password protection for these things? No. And do you… and as… and that wasn’t… that was just, like, 10% of the list, and they have 21 engineers that are working on that. There’s no way that those 21 engineers cannot be reallocated to have a bigger impact for their company. So I think a lot of folks that started building probably, like, with clawed code, I think a lot of that is going to revert back to buying it as people realize the complexities that are… that are involved.

Erik Charles:

Yeah, the opening’s easy. Alright, we’ve got about 3 minutes left. I wanna hear your forecast. you know, what’s the, you know, this summit, you know, HSE has put this together, they’re doing a great job. It’s, you know, this layer that turns a model into something that business can run on. We’re seeing the changes already. I mean, all four of us represent a change in this marketplace, and we’re starting to have the ripple effects. But, you know… it’s December 31st, 2027, or maybe we can just call it September of 2027. It’s a year from now, I get to… I somehow convince, along with HSC, to reconvene this panel.

What’s changed in one year? Who wants to go first?

Adit Jain:

Well, our revenue has grown 7 times.

Erik Charles:

Oh!

Amanda Kahlow:

Your revenues? That’s funny.

Erik Charles:

There we go. We got, we got a 7X revenue jump.

Amanda Kahlow:

Ugh.

Jaleh Rezaei:

Yeah. I mean, I think a really obvious one is, I think everyone in the company will have, An assistant that can work as effectively as, like, a full-time employee that you have hired as an assistant? We definitely see we’re close to that on the AE side. That we still need some more improvements in the model, but I can’t imagine not being there, and I think for a lot of roles. It might be a general purpose assistant. If it’s a complex role, it’ll probably be a specialized assistant, but I think everybody will have a lot more help to be able to double, triple, quadruple their productivity.

Amanda Kahlow:

Yeah, I would… I would say it’s around, the… I mean, I go back to, like, go-to-market, and I think of, like, quotas are gonna go up, because… and there’s gonna be more selling time, more face time in the field, because all the workflows around it will be taken. And then you’re… I think you’re gonna see a big delta between… like, good reps and, like, managing to the middle, or are we managing, like, the top echelon? And the top echelon are gonna basically have these massive territories?

Erik Charles:

Yeah, I think we will definitely see sales covering more. My prediction is we’re going to see the variable incentive get pushed further in the organization, especially in uncertain economic times. Unfortunately, to freeze salaries and put more into variable. the agents… AI is making it possible to measure more people on an objective basis. I’m already seeing it, I’m… I’m doing huge deals where only 30% of the seats, I’m still using seats, admittedly, here, are commissionable reps, and the rest are other employees on MBO and KPI plans, but those MBOs and KPIs, people are just getting smarter about bringing them all together.

While I wait for Julia to jump back in, very quickly, 30 seconds, how do people get ahold of you, and who should reach out to you? Jaleh, you first.

Jaleh Rezaei:

LinkedIn would be great.

Erik Charles:

Chase you down on LinkedIn. Amanda?

Amanda Kahlow:

Linkedin, but I don’t respond to any DMs, so, you know, you can follow me, or it’s just impossible, so if you really want to get to me, you gotta… and this is what I said, actually, I was gonna do a post on this. If you want to get to anyone, like, you want to… you need to find somebody who knows that person, and give real value, and ask that person to make an intro. Like, the cold emails just don’t work.

Erik Charles:

Yeah, I can see that. Adit?

Adit Jain:

Yeah, LinkedIn is the best.

Erik Charles:

Fantastic. Same. Y’all can find me on LinkedIn, or if you’re in the Denver area, ping me, because I’m actually going to be speaking Thursday morning at Denver University on sales performance. So, thank you. Julia, to you.