PRODUCT DESIGN — USER EXPERIENCE — ACCESSIBILITY — USABILITY TESTING

AI-Enabled Interview Experience

Gathering feedback at scale was important for customers. This project entailed branching the existing survey features into an adaptable, AI-enabled experience called “Flex” to gather richer feedback.

  • Create a new feature to gather direct voice-of-the-customer feedback in a scalable, low-cost way for enterprise companies.

  • Build upon and update an existing, yet outdated framework for a modern feedback delivery experience.

  • I was the primary designer for this feature, working closely with product management and engineers over each iteration of the project.

IDENTIFYING THE PROBLEM

Surveys Weren’t Enough

Our customers were hesitant to adopt our survey feature, due to the static and less engaging nature of the feedback. They much preferred our live interviews, which gave rich feedback on account of a human interviewer being able to pivot. Additionally, they were able to hear directly from the voice-of-the-customer what their thoughts were, their tone, etc. Live interviews were more expensive, inhibiting customers from scaling how much feedback they could analyze.

Problems identified from discovery:

  • Static and often short answers from survey feedback
  • Expensive human-to-human interview processes
  • Limited data causing customers to not be able to pull trends out easily

Research methods and data used:

  • Qualitative interviews from customers (existing business)
  • Downward trending surveys adoption (internal data warehouse)
  • Anecdotes about updated, lackluster pricing model (new business)

Proposed solution:

A new, asynchronous feedback collection method “Flex interviews” to fit between the high quality, but expensive Live interviews and the cheap, but surface-level Survey feedback. Flex interviews differentiators would be:

  • Video questions: Allow for questions to be recorded and auto-play to give questions more substance and require participants to answer in video or audio format, so customers could hear emotion and sentiment, and use quotes for marketing and case studies,.

  • AI Follow-ups: Upgrade from static questions to adaptable follow-ups. AI would analyze the previous video/audio answer and identify if the could deep deeper for better insights.

MVP SCOPING

Optimizing Time-to-Customer Value

A key challenge was launching the MVP quickly so we could start gathering customer feedback to iterate. I identified which components and conventions that could be reused vs. were net new and needed to be accounted for in the PRD:

New:

  • Separate pages “Flex Interviews” and “Surveys” to preserve each as distinct feedback collection methods

  • Video-recording question type (Open Ended) that supports recording questions in the builder and participant answers.

Reuse:

  • Admin-facing survey builder interface to reuse branding customization and existing questions

  • Participant-facing “Intro”, “Incentive Contact”, and “Thank you” screens

USER EXPERIENCE

Adaptable AI Follow-ups

To bolster the Flex interview experience, our next iteration was to enable AI follow-up questions. Once a participant answers, AI would determine 1. If a follow-up question is needed, and 2. What the follow-up question should be.

How to generate AI questions quickly, for participants who are already investing more than they get back?

We immediate ran into technical limitations, causing the question generation to be 7-9 seconds. In this situation, it was a non-starter to reduce the time according to our Response Time Standard documentation (see right). Advocating for the user’s experience was important, so our team worked on lowering the time.

We were able to get down to 3-5 second wait time, which was better, but still needed some clear feedback.

Response Time Standard, AI Features

While AI capabilities are powerful, they can become burdensome when poorly implemented or don't meet users' expectations.

Short Wait, 1-3 seconds

Use indeterminate loaders.

Average Wait 3-7 seconds

The maximum of 7 seconds is acceptable only for complex processes, like large data analysis or visualizations.

Long Wait, 7+ seconds

For most purposes, AI features should not be above 7 seconds. If technological limitations restrict reducing this time, refer to these suggestions:

  • Allow the user to navigate away, but notify them once the process is complete
  • Use secondary and/or tertiary animations to engage the user during a tedious wait
  • Give the user tasks or interactions while they wait

Design Decision 1: Modal with clear determinate loader

Once an Open Ended question was submitted, a modal overlay appears with a determinate loader with a progress bar and exact percentage was used to reassure users during the up-to-5-second wait.

Design Decision 2: Branded gradient mesh

For the AI Follow-up question, a branded gradient mesh was used to differentiate from the other video questions. I found a lightweight, TypeScript animation for engineering to use. I focused on keeping the copy generic: “…while we consider the best follow-up question” as there would be a chance AI would determine no follow-up was needed. This was paired with establish conventions like our AI sparkle icon and Like/Dislike feedback loop.

View the full flow prototype

OUTCOME

Iterative Testing

Customers mentioned a quality increase in insights compared to the surveys. Being able to hear the customer’s voice from the video increased the level of detailed answers.

Naturally with the MVP launch, the product team actively aggregated and planned agile investments to improve the experience and participant completion rates.

26 -> 34%

completion rate over 1 month alone, by improving and iterating on the user's experience

from direct customer feedback

Investment 1: “No audio detected” error states

We saw a handful of submission with no audio. These were contributing to the completion rate positively, but couldn’t contribute to our customer’s insights and had to be manually excluded from the completion rate. To prevent these empty submissions, we implemented an alert if no audio was detected after 3 seconds. We saw a stark decrease in empty submissions weeks after launch.

Investment 2: Fewer button clicks

A common theme in user feedback was how many clicks it took to record a question. This is rather vague feedback, so after following up with users, we clarified these user experience improvements:

  • Once the user has selected “Video” or “Audio”, automatically show their choice to prevent the mental friction of selecting an option.

  • Upon clicking their answer, users had to previously then “Start recording”. This was replaced with a countdown.

These solutions didn’t eliminate clicks (as the user feedback phrased it was), but reduced the perceived clicks and burden users felt they had to do to answer a question. Users reported immediate improvement in the experience.

Investment 3: Navigation “Back” and “Next” buttons

Originally, we launched Flex interviews as a very linear process for simplification. We found a group of users weren’t able to complete the interview due to reasons like poor internet timing them out, being pulled away for other responsibilities, etc. Once the page reloaded, users were frustrated that their answers weren’t saved and had to restart the whole interview. To fix this:

  • We started temporarily saving partial responses so participants didn’t have to restart.

  • We added “Back” and “Next” buttons, so participants could revise answers freely. This helped if they were booted to a question unexpectedly, they could return to their desired progression point.