Back
Heuristics of Google's SGE
Human-centric A.I.
Evaluating A.I. Systems: Google Generative Search
This study on Google's generative AI feature, SGE (Search Generative AI), in Google Search emphasizes a safe user experience. A heuristic evaluation found 23 violations out of 45 heuristics, highlighting issues in system visibility and user training. Testing with SGE-unfamiliar participants revealed challenges in understanding and limited engagement. The findings underscore the need for improved usability, user training, and clearer interfaces for new users.
Team
Justin Catalano
My role
UX Researcher
What is SGE
The product of a competitive race
Google introduced SGE (Search Generative AI) in 2023 as a response to the recent release of generative AI.
Google holds an 83.41% share of the global search market (Statista), emphasizing the need for safe, easy, and understandable interactions for the mass population.
SGE aims to supercharge search, providing AI-powered overviews for quicker understanding of topics, with the ability to easily follow up.
Human involvement
New experience, old users. What are some of issues that SGE could be facing?
Mental Models
Human-Centric
System Control
Heuristic analysis
Self-guided heuristic evaluation for SGE included:
• 15 Interaction Heuristics (Nielson)
• 15 Principles of Display Design (Lee)
• 15 Automation Design Principles (Billings)
• Evaluating what a full 0-1 search experience was like.
• Violations will be assigned a severity rating based on the frequency, impact, and persistence of problem.
Where and what?
Out of the 45 heuristics tested, 23 violations occurred:
• 10 issues in 15 Interaction Heuristics (IH)
• 7 issues in 15 Display Design Heuristics (DD)
• 6 issues in Automation Design Principles (AD)
Usability Testing
Finding validation through usability testing.
• Selected 5 participants unfamiliar with Generative Search through a screening questionnaire with consent.
• Conducted a pre-task questionnaire to understand participants' background with Generative AI and Google.
• Instructed participants to use Generative Search to find 3 facts about each stage of a butterfly's life cycle.
• Utilized think-out-loud process to observe participant actions, thoughts, and behavior.
• Administered Systems Usability Score (SUS) questionnaire and post-task questionnaire.
User's perspective
Finding validation through usability testing.
• Selected 5 participants unfamiliar with Generative Search through a screening questionnaire with consent.
• Conducted a pre-task questionnaire to understand participants' background with Generative AI and Google.
• Instructed participants to use Generative Search to find 3 facts about each stage of a butterfly's life cycle.
• Utilized think-out-loud process to observe participant actions, thoughts, and behavior.
• Administered Systems Usability Score (SUS) questionnaire and post-task questionnaire.
New requirements
The assessment involved interaction with the playlist categorization and equalizer accessibility features, with users providing feedback based on their overall experience in alignment with Nielsen's principles.