The Trade-off between Automation and Human Quality Assurance in Customer Support

As teams start to integrate more automation solutions into their customer support processes, they are faced with a trade-off between the speed and efficiency that automation provides and the need for human quality assurance. When it comes to sending replies out quickly, automation allows for a significant speedup in the response time, but this also means that there is less time available for human agents to review and approve each response.

For example, if an AI model flags up a conversation that is starting to go south or getting kind of hairy, where should the human agent's time be best spent? Should they intervene at every opportunity, or can the human leverage their expertise to identify when to step in and make a difference? The answer lies in finding the right balance between automation and human quality assurance. While it may seem counterintuitive to spend less time on routine back-office tasks and more time intervening in conversations that are starting to go wrong, this approach allows human agents to focus on providing exceptional customer service.

The key to making this work is to treat human customer support agents as experts who can be leveraged to make a difference. By automating many different parts of the workflow and compressing the total time down, it becomes possible to spend more time in specific nodes where human intervention is needed to ensure high-quality responses. For example, if an AI model requires human supervision for 50% of its interactions, but only needs to be intervened in 5% of those cases, then there is no need to introduce a second layer of automation.

In fact, research has shown that there is a threshold below which it's better not to have human intervention. If less than 10% of the responses are incorrect, then the AI model can effectively handle the workflow on its own without the need for human oversight. However, if more than 10% of the responses are faulty, then introducing an additional layer of automation can help ensure that only high-quality responses reach customers.

This has significant implications for how we design our customer support workflows and how we train our human agents to work with AI models. By recognizing the importance of finding the right balance between automation and human quality assurance, we can create more efficient and effective customer support processes that provide a better experience for both humans and machines.

The Power of Generative AI in Customer Support

One area where generative AI is particularly well-suited is in monitoring conversations and identifying opportunities for human intervention. Unlike traditional machine learning models, which are typically limited to recognizing patterns and anomalies, generative AI can learn to mimic the language and tone of human agents, making it easier to spot when a conversation is starting to go wrong.

In addition, generative AI tends to be less prone to empathy fatigue and approval bias, which can lead humans to become desensitized to certain types of responses or conversations. By leveraging the strengths of generative AI, teams can create more effective workflows that provide a better experience for both customers and human agents.

Real-World Applications: HighQ and its Customers

HighQ is a company that has been at the forefront of developing tools and technologies that help customer support teams work with AI models to improve efficiency and effectiveness. By providing platforms and solutions that enable teams to automate routine tasks and focus on high-touch, high-value interactions, HighQ aims to make it easier for companies to provide exceptional customer experiences.

As we discussed earlier, HighQ's platform allows customers to monitor conversations in real-time and intervene when necessary. But the company is also exploring new ways to leverage generative AI and other technologies to create more effective workflows that balance automation with human quality assurance.

Demetrios' Insights on CX Leadership and AI Adoption

Our conversation with Demetrios, who leads a team at HighQ, provided valuable insights into the challenges of balancing automation with human quality assurance in customer support. When asked about his thoughts on the trade-off between speed and human oversight, Demetrios highlighted the importance of finding the right balance.

"The key is to treat your customer support agents as experts who can be leveraged to make a difference," he said. "By automating many different parts of the workflow and compressing the total time down, it becomes possible to spend more time in specific nodes where human intervention is needed."

Demetrios also emphasized the importance of recognizing when to introduce additional layers of automation. "If less than 10% of the responses are incorrect, then the AI model can effectively handle the workflow on its own," he said. "But if more than 10% of the responses are faulty, then introducing an additional layer of automation can help ensure that only high-quality responses reach customers."

Finally, Demetrios highlighted the importance of leveraging generative AI to monitor conversations and identify opportunities for human intervention. "Generative AI is particularly well-suited for this task," he said. "It can learn to mimic the language and tone of human agents, making it easier to spot when a conversation is starting to go wrong."

Resources for Getting Started with HighQ

For teams looking to get started with using HighQ's platform or leveraging generative AI in customer support workflows, Demetrios offered some valuable advice. "Start by automating routine tasks and focusing on high-touch, high-value interactions," he said. "Then, use the insights from your conversations to inform your workflow and make adjustments as needed."

HighQ is also providing a range of resources for customers who want to learn more about its platform and how it can be used to improve efficiency and effectiveness in customer support. From blog posts and webinars to case studies and whitepapers, HighQ's website is a wealth of information on topics ranging from the benefits of automation to best practices for leveraging generative AI.

By recognizing the importance of finding the right balance between automation and human quality assurance, teams can create more efficient and effective customer support processes that provide a better experience for both humans and machines. With tools like HighQ and generative AI leading the way, it's possible to build workflows that are faster, cheaper, and more effective – while also providing exceptional customer experiences.

"WEBVTTKind: captionsLanguage: enwelcome back to the AI Minds podcast this is a podcast where we explore the companies of Tomorrow built with AI top of mind I am your host demitrios and this episode is brought to you by Deep Graham the number one text to speech and speech to text API on the internet today trusted by the world's top conversational AI leaders startups and Enterprises like twilio Spotify NASA and City Bank we are joined in this episode by Liz the founder of highq how are you doing today Demetrius thank you for having me doing well what about yourself I am great I love this energy that you're bringing to the conversation I know that we just talked uh at length about what you've been up to at highq and I want to get into the inspiration behind the product the product self but you have a bit of a backstory that I will do a little bit of a tldr on and get people up to speed so that they know you were at well born and raised in Texas and then went to MIT then said all right MIT was great but I'm going to go travel the world a little bit you were doing Commodities trading is that it yep physical Commodities trading applied for the job in New York they offered it to me and Geneva Switzerland and I said yeah let's go let's go see what it's all about not a bad gig I could see how that could be fun and then went to Singapore and did a little bit more of that but you stopped at some point doing the Commodities training what made you pivot out of that so one of my closest friends had just started a startup out in San Francisco we went to MIT together and he said come out and work with me and I loved going from a company of 10,000 people large trading company company of 10 people and that kind of kicked off the startup bug for me right I mean a lot of people will say you know you're in Customer Support automation now what does that to do with physical Commodities treating and the fact is physical Commodities treating is like 10% macroeconomics 90% process optimization shipping metal around the world selling it for a profit and that actually very much informs the way that we think about customer support oh fascinating okay so you got the startup bug then you applied to YC and you got into YC in 2017 and started doing something that I think a lot of people were doing at the time but it fizzled out right there was the year of it was the last time we had the year of the chatbot what happened I think people had rediscovered NLP and people are going around saying well what can we hit with the NLP hammer and they think customer support is always very tempting because of the sheer amount of data text and voice that you generate there and we looked at that and we said okay we do believe that automation is very much part of the future of customer support but 2017 conversational AI is not ready for prime time so what can you do what's the backend workl automation that you can build and how do we learn about that I mean we were a bunch of you know MIT nerd who went through IC who didn't know a lot about customer support and so we ended up going on this journey over the next 5 years where we essentially built like a really big Outsource contact center like hundreds of seats out in West New York and really learning from the floor up what does it mean to actually run customer support programs at scale so you went through the tech accelerator YC and then you said which I've heard they tell you to go and talk to customers you said all right you know what's better than talking to customer becoming our own customer exactly oh that is classic and so what did you learn over those years of building this company we learned a lot right because customer support is often where edge cases happen customer support by nature is a problem with a lot of service area because you know what if the customer went through everything perfectly they wouldn't have ended up in Customer Support generally maybe there's some practive Outreach things and so we said you know what if the goal is to learn all about customer Sport and automate that and that involves scaling a bit call center fine let's do it let's learn from that and then 2021 2022 hit and we kind of looked up and we said you know what conversational AI we all saw this with Chad GPT is kind of getting ready for prime time so we went back Revisited our thesis and we ended up starting to automate very systematically our entire customer support Outsourcing function and what that looked like was breaking apart all the customer support processes into literal processes and it going through and saying what can we automate what can we not automate in a very systematic fashion and then we started to automate customer support fractionally and then in some cases entirely we started to realize that all of the sort of management and quality infrastructure we built up when we were hundreds of Agents didn't work anymore right so when you have hundreds of customer service agents you have team leads you have managers and you also have like QA analysts right their job is to every week look and go and grade somewhere between 1 to 5% of last week's contacts so you can give feedback your agents and help your agents improve and also report back and say what is the quality of support that all my agents are providing and it's slow it's delayed it's a very manual audit process right between that and you might also ask your customers how you're doing like instead of sead survey between these two you get maybe 5 to 15% of data coverage out of that and you know it's not ideal to have data on a weekly delayed basis but when you're managing big teams of hum hum humans that you that are smart that you've hired for attitude humans can figure it out right it's okay to be a little bit slow but we realize that when we started to actually automate customer support you needed much faster ways to get instant feedback on quality monitoring and observability so you sort of see in real time I've got all these automations going are they going off their rails or are they not and so the first version of what is now High Q was really built for our own customer support team as a way to say how can we AI not just to directly automate customer support but to provide real time visibility and data information back to CX leaders so it's it's that monitoring layer and I do really like that because if things are going off the rails you want to know as fast as possible you can course correct yeah yeah exactly before your customers know ideally because they shouldn't have to be the ones that are reporting it to you but I do think they're taking a step back and going to these the processes part and recognizing what can we automate what is not automatable is that a word for some reason when I said it slow I don't know if it actually is a word but the part that I wanted to ask was around what were some pieces of these building blocks that you realize this is really not we cannot bring automation to this part of the process you know it's hard to say we're going go from zero to one like no automation to fully autom at right but when you start to really take a systematic process process oriented approach to what actually goes into a customer service conversation you start to realize that there are a lot of pieces that actually make maybe more sense to automate than to necessarily have a human do and we approached this from the perspective of look automation is not the goal right no one's giving you a goal D how much automation you're using what you care about are your ultimate business metrics that might be that might be first response time that might be quality that might be seat whatever that is that's what matters and so whether you use automation or not really goes towards supporting that and so when we first started what we did was we broke down a lot of um customer support interactions into the front end generally the conversational layer and then the back end like the worklow automation layer right so we've all been e-commerce customers so here's a really simple example right imagine you know you are emailing in a customer service team and saying hey I bought this it's broken please honor the one warantee right the conversational layer is the company understanding that hey that's what the customer is asking for right the backend layer is then going and say can we hook into the e-commerce system can we look up the customer can we locate their order is the with and the warranty oh they sent us a photo is that photo in fact of a broken item processing the replacement and then sending that back to the conversational aspect and so when we look at that we say okay well the conversational aspect is kind of 10% of the work the company needs to do do but it's 90% of the customer's experience right so that's where especially when it's customer facing we want to be extra extra careful and have really strong guard reils at any sort of automation we Ed but the backend processes looking things up determining whether it's qualified that automate humans are not great at clicking around doing math all of that stuff but we do use a lot of both generative models as well as other smaller ml machine learning models in our system and there when something's customer facing we sort of think about this Matrix of whether we can automate and how much to automate right because you can do many things with models you can feed it all of your support documents and policies and say please answer these questions or you can use it more as sort of a final filter in the warranty case what we would be doing is doing all the backend actions and collecting essentially a list of what was done and then you're asking a generative model to hey don't make things up from scratch this is the statement of facts reframe it for us and reframe it so that the you can give it to the customer in a way or is it reframe it just to make sure that everything that was done was correctly done ah great question two things right so you both want to think about automating directly but then we we sort of want automation first or we started automating the reply for example so here's what the customer wrote here's the statement of facts that we did please generate you know a really empathetic reply back to the customer right so we want automation first but then as we started to hit this Tipping Point of automation we realized wait before we automate anymore we actually need to stop and start thinking about quality and then that's where we started building out additional almost like validator models throughout our workflows that would then take that and say okay well let's just double check this is the reply that either the automation or the agent Des send please make sure that it's relevant based on what was said it's uh compliant with the policy and processes and then if the message is saying a replacement was generated tap the Shopify integration and make sure a replacement was actually generated so a little bit of both um our current view is that the quality monitoring is almost Step Zero before you can start to think about using a bunch of different automation Solutions you really want to make sure that your entire customer support program is instrumented for Real Time visibility so I do like this Step Zero of before you do anything before you're even touching and trying to bring automation to it let's just monitor what's happening right now what are the Baseline metrics that we can go off of and then you can say all right well this could be better it seems like there's bottlenecks here it seems like there is things that continue to happen in this part of the business or whatever it may be or with this task specifically that's where we can try and bring some kind of better methodology whether that's automation or just like having another human in the loop or having something better happening there so you're able to see the blind spots and I do think that one of the key things that you said is it doesn't really matter how you fix things you just want to make sure that you're paying attention to the metrics and that you're fixing them or you're getting whatever can help you to hit those metrics whether it's automation or bringing more Firepower into the equation with humans then use that right and yeah figure out a way to make that work so what does this like layer of just monitoring look like on the business and specifically on the customer support side of things how do you go about monitoring and then how do you go about suggesting upgrades I love that and it's really that what you everything you said was just this Focus first on understanding right the current data of the CX program the layout all in different pieces because then that allows you to one zoom out and say okay where are my current gaps if I'm going to roll out an automation or AI solution where is it going to have the most impact and then because you have all the instrumentation once you roll it out it also allows you to monitor it and see what's happening right and so what does that mean tactically which I think I think is your question here is we essentially integrate highq integrates in directly with um customer support CRM systems like zenes customer Salesforce gladly gorgeous and we essentially right along so you integrate your CRM and then we write along and as customers write in or contact the company we're categorizing what the customers are writing in about and then as the brand sends replies whether they're automated or human replies we are essentially monitoring the quality on that both on the conversational layer so you know as a generally as a general human is this a good quality conversation but then also on a policy and compliance level is what's being said in line with company policies and is what's being said actually being actioned in the backend systems and then we also predict in real time customer satisfaction right so what your customers are writing in about the quality of replies that company's producing back and then how the customers feel about that and when we talk to companies that are starting to think about what do we automate first in our customer support plan this also sort of lays that groundwork because it allows these sort of have visibility into what are the biggest contact categories that maybe we are currently really good at or could improve on that are maybe either making our customers really happy or really upset because that then allows you to map well one of my biggest categories where there's a lot of room for improvement and then we can do the Deep dive of of those big categories that need Improvement where are steps that you can systematically automate and then also use this as a tool to then make sure that you're accomplishing your quality and satisfaction goal and it does feel like when you are monitoring and you're going for that understanding first what you're going to encounter with customers is that not one system looks the same and so you have to understand the system first before you can actually make recommendations have you seen that you can productize these upgrades if will or is it something that each one you have to go in and work with the specific system to make a very custom solution so we found that we can actually productize this pretty well because you know any sort of automation we find is sort of a combination of like rural based black and white Automation and then you know models that can pull in when things are a little bit fuzzy right so a good example of maybe a producti scalable version of this is you know you you might have a conversation where the customer is asking for a refund and the company is saying yes we've price suggested for you here's a refund right now you can use an llm to to understand what is it asking for it's asking for a refund right and then you're tapping a sort of black and white integration that you've built into Shopify to say okay well was a refund also processed so for us it's sort of the black and white of building out all the Integrations building out all the hooks right and then allowing models and llms and some generative aspects to help navigate and make sense of that oh fascinating yeah so I I think I see and understand what you're saying is that I imagine a lot of times just having the Integrations might be an upgrade but then if you throw llms and you throw generative models into the mix they're understanding much more as this data is Flowing from point A to point B and they can help monitor that and potentially intercept or make it better upgrade it as it goes across I mean the other B here which I think touch on here is also the we think a lot about risk management and sort of the ROI versus the risk right um because one way of perhaps building customer service automation is to build all the integrate everything in right feed the model your support docs and say go for it right customer saying this you have the ability to process a refund and Shopify or or replacement or whatever go for it and that's that's you know a high-risk application right a lot of things like price adjustments and whether or not a customer qualifies for them that's a very deterministic thing so using a probabilistic model to do that is not only a waste of compute but also kind of scary because the amount of price adjustment a customer needs is not probabilistic it's deterministic but but when you're thinking about something like observability in QA right much much safer you can let that model go live because you're not actioning things you're really using it to give you real time visibility in a way where instead of trying to prevent downside right when you're trying to when you set automation life on your customer you're kind of trying to prevent bad things from happening here you're leveraging a lot of those same models and access to instead catch things and look for opportunities to improve so you can be a lot more I think liberal with what you allow a model to do because it's functioning in observability rather than in Direct Customer actions and this may be getting a bit into the weeds too much but I had to ask I I was thinking about that the deterministic versus non-deterministic modes that you get where you can have a big problem like the one that we've been seeing it's like companies end up in the headlines for all the wrong reasons whether it is Air Canada not giving refunds or it's the Chevy dealer who's selling uh Chevy for $1 talking about how much better tetla is than Chevy those are reasons that you don't want to have ai interacting with the end user right and Y so it is good to be thinking about that but there are as you were saying times when it's not like there's a gray area and so having these generative models produce the response to the customer or to anybody it's not the best it's not the the hammer that you you need or it's not the tool that you need to to make that happen and so are you using things like knowledge graphs in those areas are you trying to just give like a relational database answer that is pulled out and how does that look so the way that our workflows are structured is we have a lot of nodes that are deterministic nodes that tap other tools right so I me I guess the canonical example here is maybe that LMS are really bad at arithmetic right yeah right because arithmetic is a deterministic task not a probabilistic task what you actually needed to do is recognize that what you're trying to do here is a deterministic task and like tap the calculator node right and then accept that back in as part of your broader action so we think a lot about it in that sense right the step the the task is not solve current task using any sort of model the task is actually solve the current task by breaking it apart into its constituent components figure out which components are deterministic which ones are fuzzy fuzzy nodes are nodes where there are many correct answers right and then route them to the correct tools that tends to be how we think about it right so calculating a price adjustment deterministic task that's a calculator right but then communicating that back to the customer or determining whether or not it was a good communication there are many acceptable ways to tell a customer that they got a price adjustment oh fascinating okay I like it and now going back to the monitoring piece because I do really like this idea of you get to know before the customer knows when the automations aren't working as well how does that work out how do you flag things and like do you shut down the system so that if there is any doubt that things are going off the rails you don't let it go off the rails first yeah so we built highq for ourselves we essentially had that so maybe to rip off the example you gave there where you know if if for whatever reason you that you Traverse the the workflow and you generator generated a reply that's doesn't follow policy that's I don't know selling cars for a dollar right we would have an additional validation node between that and then actually sending the reply back through zenes that basically says hey this is the reply that we're about to send please make sure it's a compliance with our policies right and if no that's a great opportunity to like flush it back and send it to a human for review before you send it out um for highq when we work with um outside clients it's does still go out but as soon as the reply is sent out hiq runs and we'll check to make sure there's a good conversation and also grade it on policy and compliance so it's sort of like having a QA an AI QA analyst sitting next to each one of your agents to make sure they're doing a good job so you can catch things as soon as they go out if it's not a compliant reply yeah CU I imagine that there is a bit of a tradeoff between getting something out getting out that response and speaking to someone as quickly as possible or appeasing their whatever their complaint is when they're coming into the company versus making sure that it is the highest quality and it is following everything that it needs to follow yeah that's fair and I think that's what we sort of think about as the tradeoff between NE qual like QA and customer satisfaction at times right so we actually very much predict both because QA and seat don't necessarily move and sink right you can like hand our refunds left and right you won't have a very good QA score but you might have a really great satisfaction score and vice versa um but when it comes to that tradeoff as teams start to integrate more and more automation solutions that allows them to already greatly speed up the speed at which they're sending replies out and so that then brings the opportunity of okay well given that we can have the ability to go so quickly where do we put our human agents to make the most of their time and ability right like we have 50 customer service agents where is their time best spent it's probably not best spent processing you know routine back office tasks is probably best spent when a model Flags it up and goes hey this conversation is starting to go south or hey this conversation is getting kind of hairy can you can you plop in there right so yes there's that trade-off of time but I think it's much more about if you treat your customer support agents really as experts where can you best leverage their time yeah I've been fascinated with this question for years and I think that is probably one of the most interesting pieces is if you are going to be automating things and if you are going to have the ability to let's just say let AI do the whole workflow end to end where are you putting the human in the loop to make sure that the quality is high and to and I do like this other piece where you're saying well you know if you can automate many different parts of this then you're compressing the total time down and so it's okay if you put a human in two or three times in this Loop because still going to be doing something faster than if a human were to do the whole thing definitely and to add on to that is there's also we realized this firsthand but humans are very human so you might assume that it's always better to have a human in the loop at a specific node but what we found was that there's actually like an error rate below which it's better to not have the human so imagine you're a human and you're supervising this one automation node right if that node requires you to intervene 50 % of the time great that's a great use of you right but if the automation is good enough that you only need to intervene 5% of the time you're not going to catch that 5% you'll actually get so used to just rubber stamping it and clicking approved that you don't catch the 5% error so we realized as we went through that there's actually like a threshold where if like less than 10% is actually wrong you need to introduce a second layer of automation because you can't present that to a human they will just rubber stamp it wow that is such a cool learning and I see myself in that human category if I'm getting 95% of the things that I'm seeing are going to just be yeah okay looks good to me then I probably am going to say everything looks good to me yeah you're human right you just zone out a little bit um and that's where I think generative AI tends to be best um because they don't have things like empathy and approval fatigue yeah well there this has been absolutely fascinating talking to you I appreciate you coming on here and really going deep into the Weeds on what you're doing at highq and how you're leveraging AI to monitor you're also making sure that in these contact centers and in in these like customer support support use cases your customers are getting the most out of each interaction and so it's cool to see I'm rooting for you and want to make sure that next time I reach out to a company I hope they are using your tools so that it can ensure a good experience from my end well Demetrios thank you so much for having me and yeah we're definitely excited it's all about QA at the speed of automation there we go so if anybody wants to get in touch or start using highq where should they go to find you hi q. CX reach out there's a demo video on there or you can reach out to our team we would love to love to chat and hear more about how you're automating and thinking about autom meting QA and I will just mention this for anybody that is an avid podcast listener you all do a podcast weekly on LinkedIn yep every other Wednesday on LinkedIn we do a plane speak where we talk to mostly CX leaders not so much at just a conceptual level but really deep down into the weeds how they're leveraging AI immediately to make a difference for their teams and their customers so join us if you have a few minutes on Wednesdays excellent yeah we'll leave a link to that in the show notes and this has been awesome thank you thank you so muchwelcome back to the AI Minds podcast this is a podcast where we explore the companies of Tomorrow built with AI top of mind I am your host demitrios and this episode is brought to you by Deep Graham the number one text to speech and speech to text API on the internet today trusted by the world's top conversational AI leaders startups and Enterprises like twilio Spotify NASA and City Bank we are joined in this episode by Liz the founder of highq how are you doing today Demetrius thank you for having me doing well what about yourself I am great I love this energy that you're bringing to the conversation I know that we just talked uh at length about what you've been up to at highq and I want to get into the inspiration behind the product the product self but you have a bit of a backstory that I will do a little bit of a tldr on and get people up to speed so that they know you were at well born and raised in Texas and then went to MIT then said all right MIT was great but I'm going to go travel the world a little bit you were doing Commodities trading is that it yep physical Commodities trading applied for the job in New York they offered it to me and Geneva Switzerland and I said yeah let's go let's go see what it's all about not a bad gig I could see how that could be fun and then went to Singapore and did a little bit more of that but you stopped at some point doing the Commodities training what made you pivot out of that so one of my closest friends had just started a startup out in San Francisco we went to MIT together and he said come out and work with me and I loved going from a company of 10,000 people large trading company company of 10 people and that kind of kicked off the startup bug for me right I mean a lot of people will say you know you're in Customer Support automation now what does that to do with physical Commodities treating and the fact is physical Commodities treating is like 10% macroeconomics 90% process optimization shipping metal around the world selling it for a profit and that actually very much informs the way that we think about customer support oh fascinating okay so you got the startup bug then you applied to YC and you got into YC in 2017 and started doing something that I think a lot of people were doing at the time but it fizzled out right there was the year of it was the last time we had the year of the chatbot what happened I think people had rediscovered NLP and people are going around saying well what can we hit with the NLP hammer and they think customer support is always very tempting because of the sheer amount of data text and voice that you generate there and we looked at that and we said okay we do believe that automation is very much part of the future of customer support but 2017 conversational AI is not ready for prime time so what can you do what's the backend workl automation that you can build and how do we learn about that I mean we were a bunch of you know MIT nerd who went through IC who didn't know a lot about customer support and so we ended up going on this journey over the next 5 years where we essentially built like a really big Outsource contact center like hundreds of seats out in West New York and really learning from the floor up what does it mean to actually run customer support programs at scale so you went through the tech accelerator YC and then you said which I've heard they tell you to go and talk to customers you said all right you know what's better than talking to customer becoming our own customer exactly oh that is classic and so what did you learn over those years of building this company we learned a lot right because customer support is often where edge cases happen customer support by nature is a problem with a lot of service area because you know what if the customer went through everything perfectly they wouldn't have ended up in Customer Support generally maybe there's some practive Outreach things and so we said you know what if the goal is to learn all about customer Sport and automate that and that involves scaling a bit call center fine let's do it let's learn from that and then 2021 2022 hit and we kind of looked up and we said you know what conversational AI we all saw this with Chad GPT is kind of getting ready for prime time so we went back Revisited our thesis and we ended up starting to automate very systematically our entire customer support Outsourcing function and what that looked like was breaking apart all the customer support processes into literal processes and it going through and saying what can we automate what can we not automate in a very systematic fashion and then we started to automate customer support fractionally and then in some cases entirely we started to realize that all of the sort of management and quality infrastructure we built up when we were hundreds of Agents didn't work anymore right so when you have hundreds of customer service agents you have team leads you have managers and you also have like QA analysts right their job is to every week look and go and grade somewhere between 1 to 5% of last week's contacts so you can give feedback your agents and help your agents improve and also report back and say what is the quality of support that all my agents are providing and it's slow it's delayed it's a very manual audit process right between that and you might also ask your customers how you're doing like instead of sead survey between these two you get maybe 5 to 15% of data coverage out of that and you know it's not ideal to have data on a weekly delayed basis but when you're managing big teams of hum hum humans that you that are smart that you've hired for attitude humans can figure it out right it's okay to be a little bit slow but we realize that when we started to actually automate customer support you needed much faster ways to get instant feedback on quality monitoring and observability so you sort of see in real time I've got all these automations going are they going off their rails or are they not and so the first version of what is now High Q was really built for our own customer support team as a way to say how can we AI not just to directly automate customer support but to provide real time visibility and data information back to CX leaders so it's it's that monitoring layer and I do really like that because if things are going off the rails you want to know as fast as possible you can course correct yeah yeah exactly before your customers know ideally because they shouldn't have to be the ones that are reporting it to you but I do think they're taking a step back and going to these the processes part and recognizing what can we automate what is not automatable is that a word for some reason when I said it slow I don't know if it actually is a word but the part that I wanted to ask was around what were some pieces of these building blocks that you realize this is really not we cannot bring automation to this part of the process you know it's hard to say we're going go from zero to one like no automation to fully autom at right but when you start to really take a systematic process process oriented approach to what actually goes into a customer service conversation you start to realize that there are a lot of pieces that actually make maybe more sense to automate than to necessarily have a human do and we approached this from the perspective of look automation is not the goal right no one's giving you a goal D how much automation you're using what you care about are your ultimate business metrics that might be that might be first response time that might be quality that might be seat whatever that is that's what matters and so whether you use automation or not really goes towards supporting that and so when we first started what we did was we broke down a lot of um customer support interactions into the front end generally the conversational layer and then the back end like the worklow automation layer right so we've all been e-commerce customers so here's a really simple example right imagine you know you are emailing in a customer service team and saying hey I bought this it's broken please honor the one warantee right the conversational layer is the company understanding that hey that's what the customer is asking for right the backend layer is then going and say can we hook into the e-commerce system can we look up the customer can we locate their order is the with and the warranty oh they sent us a photo is that photo in fact of a broken item processing the replacement and then sending that back to the conversational aspect and so when we look at that we say okay well the conversational aspect is kind of 10% of the work the company needs to do do but it's 90% of the customer's experience right so that's where especially when it's customer facing we want to be extra extra careful and have really strong guard reils at any sort of automation we Ed but the backend processes looking things up determining whether it's qualified that automate humans are not great at clicking around doing math all of that stuff but we do use a lot of both generative models as well as other smaller ml machine learning models in our system and there when something's customer facing we sort of think about this Matrix of whether we can automate and how much to automate right because you can do many things with models you can feed it all of your support documents and policies and say please answer these questions or you can use it more as sort of a final filter in the warranty case what we would be doing is doing all the backend actions and collecting essentially a list of what was done and then you're asking a generative model to hey don't make things up from scratch this is the statement of facts reframe it for us and reframe it so that the you can give it to the customer in a way or is it reframe it just to make sure that everything that was done was correctly done ah great question two things right so you both want to think about automating directly but then we we sort of want automation first or we started automating the reply for example so here's what the customer wrote here's the statement of facts that we did please generate you know a really empathetic reply back to the customer right so we want automation first but then as we started to hit this Tipping Point of automation we realized wait before we automate anymore we actually need to stop and start thinking about quality and then that's where we started building out additional almost like validator models throughout our workflows that would then take that and say okay well let's just double check this is the reply that either the automation or the agent Des send please make sure that it's relevant based on what was said it's uh compliant with the policy and processes and then if the message is saying a replacement was generated tap the Shopify integration and make sure a replacement was actually generated so a little bit of both um our current view is that the quality monitoring is almost Step Zero before you can start to think about using a bunch of different automation Solutions you really want to make sure that your entire customer support program is instrumented for Real Time visibility so I do like this Step Zero of before you do anything before you're even touching and trying to bring automation to it let's just monitor what's happening right now what are the Baseline metrics that we can go off of and then you can say all right well this could be better it seems like there's bottlenecks here it seems like there is things that continue to happen in this part of the business or whatever it may be or with this task specifically that's where we can try and bring some kind of better methodology whether that's automation or just like having another human in the loop or having something better happening there so you're able to see the blind spots and I do think that one of the key things that you said is it doesn't really matter how you fix things you just want to make sure that you're paying attention to the metrics and that you're fixing them or you're getting whatever can help you to hit those metrics whether it's automation or bringing more Firepower into the equation with humans then use that right and yeah figure out a way to make that work so what does this like layer of just monitoring look like on the business and specifically on the customer support side of things how do you go about monitoring and then how do you go about suggesting upgrades I love that and it's really that what you everything you said was just this Focus first on understanding right the current data of the CX program the layout all in different pieces because then that allows you to one zoom out and say okay where are my current gaps if I'm going to roll out an automation or AI solution where is it going to have the most impact and then because you have all the instrumentation once you roll it out it also allows you to monitor it and see what's happening right and so what does that mean tactically which I think I think is your question here is we essentially integrate highq integrates in directly with um customer support CRM systems like zenes customer Salesforce gladly gorgeous and we essentially right along so you integrate your CRM and then we write along and as customers write in or contact the company we're categorizing what the customers are writing in about and then as the brand sends replies whether they're automated or human replies we are essentially monitoring the quality on that both on the conversational layer so you know as a generally as a general human is this a good quality conversation but then also on a policy and compliance level is what's being said in line with company policies and is what's being said actually being actioned in the backend systems and then we also predict in real time customer satisfaction right so what your customers are writing in about the quality of replies that company's producing back and then how the customers feel about that and when we talk to companies that are starting to think about what do we automate first in our customer support plan this also sort of lays that groundwork because it allows these sort of have visibility into what are the biggest contact categories that maybe we are currently really good at or could improve on that are maybe either making our customers really happy or really upset because that then allows you to map well one of my biggest categories where there's a lot of room for improvement and then we can do the Deep dive of of those big categories that need Improvement where are steps that you can systematically automate and then also use this as a tool to then make sure that you're accomplishing your quality and satisfaction goal and it does feel like when you are monitoring and you're going for that understanding first what you're going to encounter with customers is that not one system looks the same and so you have to understand the system first before you can actually make recommendations have you seen that you can productize these upgrades if will or is it something that each one you have to go in and work with the specific system to make a very custom solution so we found that we can actually productize this pretty well because you know any sort of automation we find is sort of a combination of like rural based black and white Automation and then you know models that can pull in when things are a little bit fuzzy right so a good example of maybe a producti scalable version of this is you know you you might have a conversation where the customer is asking for a refund and the company is saying yes we've price suggested for you here's a refund right now you can use an llm to to understand what is it asking for it's asking for a refund right and then you're tapping a sort of black and white integration that you've built into Shopify to say okay well was a refund also processed so for us it's sort of the black and white of building out all the Integrations building out all the hooks right and then allowing models and llms and some generative aspects to help navigate and make sense of that oh fascinating yeah so I I think I see and understand what you're saying is that I imagine a lot of times just having the Integrations might be an upgrade but then if you throw llms and you throw generative models into the mix they're understanding much more as this data is Flowing from point A to point B and they can help monitor that and potentially intercept or make it better upgrade it as it goes across I mean the other B here which I think touch on here is also the we think a lot about risk management and sort of the ROI versus the risk right um because one way of perhaps building customer service automation is to build all the integrate everything in right feed the model your support docs and say go for it right customer saying this you have the ability to process a refund and Shopify or or replacement or whatever go for it and that's that's you know a high-risk application right a lot of things like price adjustments and whether or not a customer qualifies for them that's a very deterministic thing so using a probabilistic model to do that is not only a waste of compute but also kind of scary because the amount of price adjustment a customer needs is not probabilistic it's deterministic but but when you're thinking about something like observability in QA right much much safer you can let that model go live because you're not actioning things you're really using it to give you real time visibility in a way where instead of trying to prevent downside right when you're trying to when you set automation life on your customer you're kind of trying to prevent bad things from happening here you're leveraging a lot of those same models and access to instead catch things and look for opportunities to improve so you can be a lot more I think liberal with what you allow a model to do because it's functioning in observability rather than in Direct Customer actions and this may be getting a bit into the weeds too much but I had to ask I I was thinking about that the deterministic versus non-deterministic modes that you get where you can have a big problem like the one that we've been seeing it's like companies end up in the headlines for all the wrong reasons whether it is Air Canada not giving refunds or it's the Chevy dealer who's selling uh Chevy for $1 talking about how much better tetla is than Chevy those are reasons that you don't want to have ai interacting with the end user right and Y so it is good to be thinking about that but there are as you were saying times when it's not like there's a gray area and so having these generative models produce the response to the customer or to anybody it's not the best it's not the the hammer that you you need or it's not the tool that you need to to make that happen and so are you using things like knowledge graphs in those areas are you trying to just give like a relational database answer that is pulled out and how does that look so the way that our workflows are structured is we have a lot of nodes that are deterministic nodes that tap other tools right so I me I guess the canonical example here is maybe that LMS are really bad at arithmetic right yeah right because arithmetic is a deterministic task not a probabilistic task what you actually needed to do is recognize that what you're trying to do here is a deterministic task and like tap the calculator node right and then accept that back in as part of your broader action so we think a lot about it in that sense right the step the the task is not solve current task using any sort of model the task is actually solve the current task by breaking it apart into its constituent components figure out which components are deterministic which ones are fuzzy fuzzy nodes are nodes where there are many correct answers right and then route them to the correct tools that tends to be how we think about it right so calculating a price adjustment deterministic task that's a calculator right but then communicating that back to the customer or determining whether or not it was a good communication there are many acceptable ways to tell a customer that they got a price adjustment oh fascinating okay I like it and now going back to the monitoring piece because I do really like this idea of you get to know before the customer knows when the automations aren't working as well how does that work out how do you flag things and like do you shut down the system so that if there is any doubt that things are going off the rails you don't let it go off the rails first yeah so we built highq for ourselves we essentially had that so maybe to rip off the example you gave there where you know if if for whatever reason you that you Traverse the the workflow and you generator generated a reply that's doesn't follow policy that's I don't know selling cars for a dollar right we would have an additional validation node between that and then actually sending the reply back through zenes that basically says hey this is the reply that we're about to send please make sure it's a compliance with our policies right and if no that's a great opportunity to like flush it back and send it to a human for review before you send it out um for highq when we work with um outside clients it's does still go out but as soon as the reply is sent out hiq runs and we'll check to make sure there's a good conversation and also grade it on policy and compliance so it's sort of like having a QA an AI QA analyst sitting next to each one of your agents to make sure they're doing a good job so you can catch things as soon as they go out if it's not a compliant reply yeah CU I imagine that there is a bit of a tradeoff between getting something out getting out that response and speaking to someone as quickly as possible or appeasing their whatever their complaint is when they're coming into the company versus making sure that it is the highest quality and it is following everything that it needs to follow yeah that's fair and I think that's what we sort of think about as the tradeoff between NE qual like QA and customer satisfaction at times right so we actually very much predict both because QA and seat don't necessarily move and sink right you can like hand our refunds left and right you won't have a very good QA score but you might have a really great satisfaction score and vice versa um but when it comes to that tradeoff as teams start to integrate more and more automation solutions that allows them to already greatly speed up the speed at which they're sending replies out and so that then brings the opportunity of okay well given that we can have the ability to go so quickly where do we put our human agents to make the most of their time and ability right like we have 50 customer service agents where is their time best spent it's probably not best spent processing you know routine back office tasks is probably best spent when a model Flags it up and goes hey this conversation is starting to go south or hey this conversation is getting kind of hairy can you can you plop in there right so yes there's that trade-off of time but I think it's much more about if you treat your customer support agents really as experts where can you best leverage their time yeah I've been fascinated with this question for years and I think that is probably one of the most interesting pieces is if you are going to be automating things and if you are going to have the ability to let's just say let AI do the whole workflow end to end where are you putting the human in the loop to make sure that the quality is high and to and I do like this other piece where you're saying well you know if you can automate many different parts of this then you're compressing the total time down and so it's okay if you put a human in two or three times in this Loop because still going to be doing something faster than if a human were to do the whole thing definitely and to add on to that is there's also we realized this firsthand but humans are very human so you might assume that it's always better to have a human in the loop at a specific node but what we found was that there's actually like an error rate below which it's better to not have the human so imagine you're a human and you're supervising this one automation node right if that node requires you to intervene 50 % of the time great that's a great use of you right but if the automation is good enough that you only need to intervene 5% of the time you're not going to catch that 5% you'll actually get so used to just rubber stamping it and clicking approved that you don't catch the 5% error so we realized as we went through that there's actually like a threshold where if like less than 10% is actually wrong you need to introduce a second layer of automation because you can't present that to a human they will just rubber stamp it wow that is such a cool learning and I see myself in that human category if I'm getting 95% of the things that I'm seeing are going to just be yeah okay looks good to me then I probably am going to say everything looks good to me yeah you're human right you just zone out a little bit um and that's where I think generative AI tends to be best um because they don't have things like empathy and approval fatigue yeah well there this has been absolutely fascinating talking to you I appreciate you coming on here and really going deep into the Weeds on what you're doing at highq and how you're leveraging AI to monitor you're also making sure that in these contact centers and in in these like customer support support use cases your customers are getting the most out of each interaction and so it's cool to see I'm rooting for you and want to make sure that next time I reach out to a company I hope they are using your tools so that it can ensure a good experience from my end well Demetrios thank you so much for having me and yeah we're definitely excited it's all about QA at the speed of automation there we go so if anybody wants to get in touch or start using highq where should they go to find you hi q. CX reach out there's a demo video on there or you can reach out to our team we would love to love to chat and hear more about how you're automating and thinking about autom meting QA and I will just mention this for anybody that is an avid podcast listener you all do a podcast weekly on LinkedIn yep every other Wednesday on LinkedIn we do a plane speak where we talk to mostly CX leaders not so much at just a conceptual level but really deep down into the weeds how they're leveraging AI immediately to make a difference for their teams and their customers so join us if you have a few minutes on Wednesdays excellent yeah we'll leave a link to that in the show notes and this has been awesome thank you thank you so muchwelcome back to the AI Minds podcast this is a podcast where we explore the companies of Tomorrow built with AI top of mind I am your host demitrios and this episode is brought to you by Deep Graham the number one text to speech and speech to text API on the internet today trusted by the world's top conversational AI leaders startups and Enterprises like twilio Spotify NASA and City Bank we are joined in this episode by Liz the founder of highq how are you doing today Demetrius thank you for having me doing well what about yourself I am great I love this energy that you're bringing to the conversation I know that we just talked uh at length about what you've been up to at highq and I want to get into the inspiration behind the product the product self but you have a bit of a backstory that I will do a little bit of a tldr on and get people up to speed so that they know you were at well born and raised in Texas and then went to MIT then said all right MIT was great but I'm going to go travel the world a little bit you were doing Commodities trading is that it yep physical Commodities trading applied for the job in New York they offered it to me and Geneva Switzerland and I said yeah let's go let's go see what it's all about not a bad gig I could see how that could be fun and then went to Singapore and did a little bit more of that but you stopped at some point doing the Commodities training what made you pivot out of that so one of my closest friends had just started a startup out in San Francisco we went to MIT together and he said come out and work with me and I loved going from a company of 10,000 people large trading company company of 10 people and that kind of kicked off the startup bug for me right I mean a lot of people will say you know you're in Customer Support automation now what does that to do with physical Commodities treating and the fact is physical Commodities treating is like 10% macroeconomics 90% process optimization shipping metal around the world selling it for a profit and that actually very much informs the way that we think about customer support oh fascinating okay so you got the startup bug then you applied to YC and you got into YC in 2017 and started doing something that I think a lot of people were doing at the time but it fizzled out right there was the year of it was the last time we had the year of the chatbot what happened I think people had rediscovered NLP and people are going around saying well what can we hit with the NLP hammer and they think customer support is always very tempting because of the sheer amount of data text and voice that you generate there and we looked at that and we said okay we do believe that automation is very much part of the future of customer support but 2017 conversational AI is not ready for prime time so what can you do what's the backend workl automation that you can build and how do we learn about that I mean we were a bunch of you know MIT nerd who went through IC who didn't know a lot about customer support and so we ended up going on this journey over the next 5 years where we essentially built like a really big Outsource contact center like hundreds of seats out in West New York and really learning from the floor up what does it mean to actually run customer support programs at scale so you went through the tech accelerator YC and then you said which I've heard they tell you to go and talk to customers you said all right you know what's better than talking to customer becoming our own customer exactly oh that is classic and so what did you learn over those years of building this company we learned a lot right because customer support is often where edge cases happen customer support by nature is a problem with a lot of service area because you know what if the customer went through everything perfectly they wouldn't have ended up in Customer Support generally maybe there's some practive Outreach things and so we said you know what if the goal is to learn all about customer Sport and automate that and that involves scaling a bit call center fine let's do it let's learn from that and then 2021 2022 hit and we kind of looked up and we said you know what conversational AI we all saw this with Chad GPT is kind of getting ready for prime time so we went back Revisited our thesis and we ended up starting to automate very systematically our entire customer support Outsourcing function and what that looked like was breaking apart all the customer support processes into literal processes and it going through and saying what can we automate what can we not automate in a very systematic fashion and then we started to automate customer support fractionally and then in some cases entirely we started to realize that all of the sort of management and quality infrastructure we built up when we were hundreds of Agents didn't work anymore right so when you have hundreds of customer service agents you have team leads you have managers and you also have like QA analysts right their job is to every week look and go and grade somewhere between 1 to 5% of last week's contacts so you can give feedback your agents and help your agents improve and also report back and say what is the quality of support that all my agents are providing and it's slow it's delayed it's a very manual audit process right between that and you might also ask your customers how you're doing like instead of sead survey between these two you get maybe 5 to 15% of data coverage out of that and you know it's not ideal to have data on a weekly delayed basis but when you're managing big teams of hum hum humans that you that are smart that you've hired for attitude humans can figure it out right it's okay to be a little bit slow but we realize that when we started to actually automate customer support you needed much faster ways to get instant feedback on quality monitoring and observability so you sort of see in real time I've got all these automations going are they going off their rails or are they not and so the first version of what is now High Q was really built for our own customer support team as a way to say how can we AI not just to directly automate customer support but to provide real time visibility and data information back to CX leaders so it's it's that monitoring layer and I do really like that because if things are going off the rails you want to know as fast as possible you can course correct yeah yeah exactly before your customers know ideally because they shouldn't have to be the ones that are reporting it to you but I do think they're taking a step back and going to these the processes part and recognizing what can we automate what is not automatable is that a word for some reason when I said it slow I don't know if it actually is a word but the part that I wanted to ask was around what were some pieces of these building blocks that you realize this is really not we cannot bring automation to this part of the process you know it's hard to say we're going go from zero to one like no automation to fully autom at right but when you start to really take a systematic process process oriented approach to what actually goes into a customer service conversation you start to realize that there are a lot of pieces that actually make maybe more sense to automate than to necessarily have a human do and we approached this from the perspective of look automation is not the goal right no one's giving you a goal D how much automation you're using what you care about are your ultimate business metrics that might be that might be first response time that might be quality that might be seat whatever that is that's what matters and so whether you use automation or not really goes towards supporting that and so when we first started what we did was we broke down a lot of um customer support interactions into the front end generally the conversational layer and then the back end like the worklow automation layer right so we've all been e-commerce customers so here's a really simple example right imagine you know you are emailing in a customer service team and saying hey I bought this it's broken please honor the one warantee right the conversational layer is the company understanding that hey that's what the customer is asking for right the backend layer is then going and say can we hook into the e-commerce system can we look up the customer can we locate their order is the with and the warranty oh they sent us a photo is that photo in fact of a broken item processing the replacement and then sending that back to the conversational aspect and so when we look at that we say okay well the conversational aspect is kind of 10% of the work the company needs to do do but it's 90% of the customer's experience right so that's where especially when it's customer facing we want to be extra extra careful and have really strong guard reils at any sort of automation we Ed but the backend processes looking things up determining whether it's qualified that automate humans are not great at clicking around doing math all of that stuff but we do use a lot of both generative models as well as other smaller ml machine learning models in our system and there when something's customer facing we sort of think about this Matrix of whether we can automate and how much to automate right because you can do many things with models you can feed it all of your support documents and policies and say please answer these questions or you can use it more as sort of a final filter in the warranty case what we would be doing is doing all the backend actions and collecting essentially a list of what was done and then you're asking a generative model to hey don't make things up from scratch this is the statement of facts reframe it for us and reframe it so that the you can give it to the customer in a way or is it reframe it just to make sure that everything that was done was correctly done ah great question two things right so you both want to think about automating directly but then we we sort of want automation first or we started automating the reply for example so here's what the customer wrote here's the statement of facts that we did please generate you know a really empathetic reply back to the customer right so we want automation first but then as we started to hit this Tipping Point of automation we realized wait before we automate anymore we actually need to stop and start thinking about quality and then that's where we started building out additional almost like validator models throughout our workflows that would then take that and say okay well let's just double check this is the reply that either the automation or the agent Des send please make sure that it's relevant based on what was said it's uh compliant with the policy and processes and then if the message is saying a replacement was generated tap the Shopify integration and make sure a replacement was actually generated so a little bit of both um our current view is that the quality monitoring is almost Step Zero before you can start to think about using a bunch of different automation Solutions you really want to make sure that your entire customer support program is instrumented for Real Time visibility so I do like this Step Zero of before you do anything before you're even touching and trying to bring automation to it let's just monitor what's happening right now what are the Baseline metrics that we can go off of and then you can say all right well this could be better it seems like there's bottlenecks here it seems like there is things that continue to happen in this part of the business or whatever it may be or with this task specifically that's where we can try and bring some kind of better methodology whether that's automation or just like having another human in the loop or having something better happening there so you're able to see the blind spots and I do think that one of the key things that you said is it doesn't really matter how you fix things you just want to make sure that you're paying attention to the metrics and that you're fixing them or you're getting whatever can help you to hit those metrics whether it's automation or bringing more Firepower into the equation with humans then use that right and yeah figure out a way to make that work so what does this like layer of just monitoring look like on the business and specifically on the customer support side of things how do you go about monitoring and then how do you go about suggesting upgrades I love that and it's really that what you everything you said was just this Focus first on understanding right the current data of the CX program the layout all in different pieces because then that allows you to one zoom out and say okay where are my current gaps if I'm going to roll out an automation or AI solution where is it going to have the most impact and then because you have all the instrumentation once you roll it out it also allows you to monitor it and see what's happening right and so what does that mean tactically which I think I think is your question here is we essentially integrate highq integrates in directly with um customer support CRM systems like zenes customer Salesforce gladly gorgeous and we essentially right along so you integrate your CRM and then we write along and as customers write in or contact the company we're categorizing what the customers are writing in about and then as the brand sends replies whether they're automated or human replies we are essentially monitoring the quality on that both on the conversational layer so you know as a generally as a general human is this a good quality conversation but then also on a policy and compliance level is what's being said in line with company policies and is what's being said actually being actioned in the backend systems and then we also predict in real time customer satisfaction right so what your customers are writing in about the quality of replies that company's producing back and then how the customers feel about that and when we talk to companies that are starting to think about what do we automate first in our customer support plan this also sort of lays that groundwork because it allows these sort of have visibility into what are the biggest contact categories that maybe we are currently really good at or could improve on that are maybe either making our customers really happy or really upset because that then allows you to map well one of my biggest categories where there's a lot of room for improvement and then we can do the Deep dive of of those big categories that need Improvement where are steps that you can systematically automate and then also use this as a tool to then make sure that you're accomplishing your quality and satisfaction goal and it does feel like when you are monitoring and you're going for that understanding first what you're going to encounter with customers is that not one system looks the same and so you have to understand the system first before you can actually make recommendations have you seen that you can productize these upgrades if will or is it something that each one you have to go in and work with the specific system to make a very custom solution so we found that we can actually productize this pretty well because you know any sort of automation we find is sort of a combination of like rural based black and white Automation and then you know models that can pull in when things are a little bit fuzzy right so a good example of maybe a producti scalable version of this is you know you you might have a conversation where the customer is asking for a refund and the company is saying yes we've price suggested for you here's a refund right now you can use an llm to to understand what is it asking for it's asking for a refund right and then you're tapping a sort of black and white integration that you've built into Shopify to say okay well was a refund also processed so for us it's sort of the black and white of building out all the Integrations building out all the hooks right and then allowing models and llms and some generative aspects to help navigate and make sense of that oh fascinating yeah so I I think I see and understand what you're saying is that I imagine a lot of times just having the Integrations might be an upgrade but then if you throw llms and you throw generative models into the mix they're understanding much more as this data is Flowing from point A to point B and they can help monitor that and potentially intercept or make it better upgrade it as it goes across I mean the other B here which I think touch on here is also the we think a lot about risk management and sort of the ROI versus the risk right um because one way of perhaps building customer service automation is to build all the integrate everything in right feed the model your support docs and say go for it right customer saying this you have the ability to process a refund and Shopify or or replacement or whatever go for it and that's that's you know a high-risk application right a lot of things like price adjustments and whether or not a customer qualifies for them that's a very deterministic thing so using a probabilistic model to do that is not only a waste of compute but also kind of scary because the amount of price adjustment a customer needs is not probabilistic it's deterministic but but when you're thinking about something like observability in QA right much much safer you can let that model go live because you're not actioning things you're really using it to give you real time visibility in a way where instead of trying to prevent downside right when you're trying to when you set automation life on your customer you're kind of trying to prevent bad things from happening here you're leveraging a lot of those same models and access to instead catch things and look for opportunities to improve so you can be a lot more I think liberal with what you allow a model to do because it's functioning in observability rather than in Direct Customer actions and this may be getting a bit into the weeds too much but I had to ask I I was thinking about that the deterministic versus non-deterministic modes that you get where you can have a big problem like the one that we've been seeing it's like companies end up in the headlines for all the wrong reasons whether it is Air Canada not giving refunds or it's the Chevy dealer who's selling uh Chevy for $1 talking about how much better tetla is than Chevy those are reasons that you don't want to have ai interacting with the end user right and Y so it is good to be thinking about that but there are as you were saying times when it's not like there's a gray area and so having these generative models produce the response to the customer or to anybody it's not the best it's not the the hammer that you you need or it's not the tool that you need to to make that happen and so are you using things like knowledge graphs in those areas are you trying to just give like a relational database answer that is pulled out and how does that look so the way that our workflows are structured is we have a lot of nodes that are deterministic nodes that tap other tools right so I me I guess the canonical example here is maybe that LMS are really bad at arithmetic right yeah right because arithmetic is a deterministic task not a probabilistic task what you actually needed to do is recognize that what you're trying to do here is a deterministic task and like tap the calculator node right and then accept that back in as part of your broader action so we think a lot about it in that sense right the step the the task is not solve current task using any sort of model the task is actually solve the current task by breaking it apart into its constituent components figure out which components are deterministic which ones are fuzzy fuzzy nodes are nodes where there are many correct answers right and then route them to the correct tools that tends to be how we think about it right so calculating a price adjustment deterministic task that's a calculator right but then communicating that back to the customer or determining whether or not it was a good communication there are many acceptable ways to tell a customer that they got a price adjustment oh fascinating okay I like it and now going back to the monitoring piece because I do really like this idea of you get to know before the customer knows when the automations aren't working as well how does that work out how do you flag things and like do you shut down the system so that if there is any doubt that things are going off the rails you don't let it go off the rails first yeah so we built highq for ourselves we essentially had that so maybe to rip off the example you gave there where you know if if for whatever reason you that you Traverse the the workflow and you generator generated a reply that's doesn't follow policy that's I don't know selling cars for a dollar right we would have an additional validation node between that and then actually sending the reply back through zenes that basically says hey this is the reply that we're about to send please make sure it's a compliance with our policies right and if no that's a great opportunity to like flush it back and send it to a human for review before you send it out um for highq when we work with um outside clients it's does still go out but as soon as the reply is sent out hiq runs and we'll check to make sure there's a good conversation and also grade it on policy and compliance so it's sort of like having a QA an AI QA analyst sitting next to each one of your agents to make sure they're doing a good job so you can catch things as soon as they go out if it's not a compliant reply yeah CU I imagine that there is a bit of a tradeoff between getting something out getting out that response and speaking to someone as quickly as possible or appeasing their whatever their complaint is when they're coming into the company versus making sure that it is the highest quality and it is following everything that it needs to follow yeah that's fair and I think that's what we sort of think about as the tradeoff between NE qual like QA and customer satisfaction at times right so we actually very much predict both because QA and seat don't necessarily move and sink right you can like hand our refunds left and right you won't have a very good QA score but you might have a really great satisfaction score and vice versa um but when it comes to that tradeoff as teams start to integrate more and more automation solutions that allows them to already greatly speed up the speed at which they're sending replies out and so that then brings the opportunity of okay well given that we can have the ability to go so quickly where do we put our human agents to make the most of their time and ability right like we have 50 customer service agents where is their time best spent it's probably not best spent processing you know routine back office tasks is probably best spent when a model Flags it up and goes hey this conversation is starting to go south or hey this conversation is getting kind of hairy can you can you plop in there right so yes there's that trade-off of time but I think it's much more about if you treat your customer support agents really as experts where can you best leverage their time yeah I've been fascinated with this question for years and I think that is probably one of the most interesting pieces is if you are going to be automating things and if you are going to have the ability to let's just say let AI do the whole workflow end to end where are you putting the human in the loop to make sure that the quality is high and to and I do like this other piece where you're saying well you know if you can automate many different parts of this then you're compressing the total time down and so it's okay if you put a human in two or three times in this Loop because still going to be doing something faster than if a human were to do the whole thing definitely and to add on to that is there's also we realized this firsthand but humans are very human so you might assume that it's always better to have a human in the loop at a specific node but what we found was that there's actually like an error rate below which it's better to not have the human so imagine you're a human and you're supervising this one automation node right if that node requires you to intervene 50 % of the time great that's a great use of you right but if the automation is good enough that you only need to intervene 5% of the time you're not going to catch that 5% you'll actually get so used to just rubber stamping it and clicking approved that you don't catch the 5% error so we realized as we went through that there's actually like a threshold where if like less than 10% is actually wrong you need to introduce a second layer of automation because you can't present that to a human they will just rubber stamp it wow that is such a cool learning and I see myself in that human category if I'm getting 95% of the things that I'm seeing are going to just be yeah okay looks good to me then I probably am going to say everything looks good to me yeah you're human right you just zone out a little bit um and that's where I think generative AI tends to be best um because they don't have things like empathy and approval fatigue yeah well there this has been absolutely fascinating talking to you I appreciate you coming on here and really going deep into the Weeds on what you're doing at highq and how you're leveraging AI to monitor you're also making sure that in these contact centers and in in these like customer support support use cases your customers are getting the most out of each interaction and so it's cool to see I'm rooting for you and want to make sure that next time I reach out to a company I hope they are using your tools so that it can ensure a good experience from my end well Demetrios thank you so much for having me and yeah we're definitely excited it's all about QA at the speed of automation there we go so if anybody wants to get in touch or start using highq where should they go to find you hi q. CX reach out there's a demo video on there or you can reach out to our team we would love to love to chat and hear more about how you're automating and thinking about autom meting QA and I will just mention this for anybody that is an avid podcast listener you all do a podcast weekly on LinkedIn yep every other Wednesday on LinkedIn we do a plane speak where we talk to mostly CX leaders not so much at just a conceptual level but really deep down into the weeds how they're leveraging AI immediately to make a difference for their teams and their customers so join us if you have a few minutes on Wednesdays excellent yeah we'll leave a link to that in the show notes and this has been awesome thank you thank you so much\n"