For years, the PC industry has faced a central question: what’s the ‘right’ way to test laptops?
Traditional benchmarks have succeeded in measuring raw computing power with highly repeatable tests in lab conditions. But after extensive human-centered research, Intel has found that these benchmarks don’t completely reflect the criteria that matter most to people. The right benchmarks and pure processing speed are still important, but not a complete view of the best laptop experience. People want a laptop that’s always ready, helps them focus and stays responsive as they multi-task throughout the day—all with stellar battery life.
People may care about pure processing speed only to the extent that it impacts their experience, but that does not define the full expectations of a great laptop experience. That’s why, when we launched the Project Athena innovation program in 2019, we anchored our testing approach in solving these human, real-world needs.
And it’s why we regularly update this testing methodology with each successive Project Athena specification to reflect new research on the latest trends. For example, laptops that feature the Intel® Evo™ brand must meet all the testing requirements of the 2nd edition Project Athena spec that aims to match real-world user experience needs.
Intel believes the “right” way to test laptops is by using metrics that measure real-world user experience. Testing results then reflect true user utility and value—which is ultimately what matters to people.
To help the industry deliver a new class of advanced laptops, Intel has developed a one-of-a-kind, automated testing tool that:
1. Focuses on key experiences
2. Reflects real-world workflows and conditions
3. Equips OEMs to optimize platform designs
Key Experiences That Matter
Project Athena rigorously tests designs for three foundational key experiences: consistent responsiveness, instant wake and a worry-free day of battery life.1 These experiences, drawn from ethnographic research and measured by a set of Key Experience Indicators (KEIs), provide the best basis for impactful laptop innovation.
Starting with People and Their Experiences
Project Athena’s real-world testing is based on extensive, in-depth research to identify the experiences that matter most to people, especially the ambitious and demanding people who expect the best from their laptops, without compromise:
• Human-centered innovation: Intel’s research teams boast more than 15 years of global research around human-centered innovation, helping to provide a detailed picture of how people use their laptops, what their pain points are and where innovations can help. In particular, the research focused on deep immersion sessions with people who depend on their laptops to manage their dynamic, fast-paced personal and professional lives.
• Identifying key experiences: This research revealed that people want their laptops to help them focus, adapt and always be ready and connected. These are essential to help people stay in their flow, so they can work, connect and play without interruption. To achieve this, we determined that laptops need to wake in less than a second; respond quickly and efficiently to people’s inputs; and have worry-free battery life and charge rapidly—the key experiences for Project Athena.
Defining the Key Experience Indicators (KEIs)
Intel’s technical experts then translated these key experiences into a set of quantitative KEIs—the backbone of our real-world testing methodology:
• Focusing on impactful interactions: To select the KEIs, Intel zeroed in on the tasks that have the greatest impact and are the most meaningful to a person’s laptop experience. For the second edition, which is represented by the Evo brand, Intel put a specific emphasis on the connection between local and cloud-based tasks to better reflect today’s agile working environment. Launching email, starting video calls, creating charts and loading browser videos are the kinds of essential interactions that are included in the 25 KEIs in Project Athena’s second edition.
• Technical decomposition: Drawing on more than 15 years of Intel’s user testing, Intel’s experts defined time targets—down to the millisecond—for each KEI. This testing uses MOS (mean opinion score) models to determine the exact threshold for a best-in-class experience for each interaction. For example, in Project Athena’s second edition, a laptop must be able to wake in an instant, play a YouTube video quickly or launch Outlook in a matter of seconds. Battery life must last for more than 9 hours under real-world conditions2, while being able to charge 4 hours of power in just 30 minutes with full HD displays.3
Real-World Workflows and Conditions
The Project Athena innovation program measures the KEIs with a realistic “day-in-the-life” workflow in conditions that mirror how people typically use their laptops. This provides a more practical view of performance, as opposed to using only synthetic workflows and artificial lab conditions.
A Rigorous, Realistic Workflow
Unlike benchmarks that are designed to tax specific components such as the CPU or the disk, Project Athena testing is designed to reflect a “day in the life” of a person who depends on their laptop to get things done. The testing tool repeatedly runs a workflow of more than 200 tasks, including the KEIs, which model how people use their laptops every day, based on ethnographic research and telemetry data.
This research revealed that people constantly switch between four “computing modes”: Communication and sharing; personal organization; expert production; and casual creation. This includes everything from less performance-demanding activities, like web browsing, online shopping or reading the news, to more demanding tasks, such as Zoom calls, YouTube streaming or working in the cloud with Google Suite. All of this is incorporated into the workflow with Project Athena’s second edition.
The testing tool also runs with multiple applications open at once, including constant companions like Outlook, Google Chrome, Spotify, Slack and more. This realistic concurrency tests the laptop’s responsiveness and battery life in the same way as a person who is constantly multi-tasking, browsing the Internet and streaming videos throughout the day.
To ensure the battery-life test is as realistic as possible, we use real telemetry data on communication and sharing, casual creation, organization and expert production workflows. Each test takes about two hours to complete, and we require three unique iterations of the test to ensure consistency.
Improvements in Second Edition Testing
|More people doing more multi-tasking||Added more constant companions, including Spotify, LinkedIn, Slack and Google Drive|
|Growing expectations for responsiveness||Increased number of KEIs from 15 to 25; increased the responsiveness requirements for all KEIs|
|Growing expectations for battery life||Intensified the workload to make the 9 hours of battery life more meaningful; added delays in workflow that reflect a person’s pauses during day|
|The rise of the cloud||Added KEIs for Google Drive tasks, editing Google Sheets files; added Google Drive as constant companion, along with interacting with Office files on the cloud via Office 365|
|Adoption of tools for work collaboration and personal connection||Added KEIs for Zoom tasks; added Slack as constant companion|
Percentage of Time Tested based on User-Telemetry
Project Athena tests in real-world conditions for a more accurate picture of performance. This differs from most benchmarks, which are typically measured in artificial lab conditions—for example, the laptop is plugged into the wall, with a dimmed screen and no Wi-Fi connection. Instead, with Project Athena, the laptop is tested on battery power, on a live network, with common background applications open, the latest software updates, logged into multiple cloud accounts and with the display set to ≥250 nits brightness on LCD panels.
The real-world testing tool runs 15 times to ensure a consistent result for the responsiveness KEIs. Each iteration of the workflow captures changes in responsiveness as the battery drains, and the laptop must exceed a stringent threshold to be verified. If a laptop does not meet these requirements, it undergoes further tuning by Intel and the OEM. In every workflow, the laptop is unplugged, so that Project Athena-verified laptop designs are responsive whenever and wherever they’re needed.
OEMs Empowered to Optimize Platform Designs
Project Athena’s real-world testing provides OEMs with an automated tool that they can use to assess, tune and improve their laptop designs. This equips OEMs—and the entire PC ecosystem—to deliver meaningful innovation, from balanced performance to best-in-class platform technologies.
Driving Innovation Across the PC Platform
OEMs use Intel’s real-world testing tool to optimize performance and battery life, as well as make key decisions about the entire PC platform:
• Drilling into the results: The automated testing tool enables an OEM to view its results for each of the KEIs across multiple runs. Engineers can use this information to identify pain points, understand the problem and tune the system for better performance and battery life.
• Beyond the CPU: OEMs can also use the tool to make decisions about components and the overall PC platform, because the test requires more than just processing speed. Laptops must also deliver reliable connectivity for online and cloud-enabled tasks, sophisticated memory and storage for responsiveness, and platform optimization to meet battery life requirements.
• Project Athena-verified: Ultimately, each laptop design must meet these KEIs and the spec in order to be verified as a Project Athena laptop4. This ensures that people can buy any Project Athena-verified laptop design with the confidence that it will deliver on their most important key experiences.
Just the Beginning
Project Athena and its unique approach to real-world testing is already helping the ecosystem create complete PC platforms tailored for people’s needs. These laptop designs are tuned, tested and verified to deliver on key experiences under real-world conditions, with a comprehensive set of KEIs that measure performance to the millisecond.
Intel will continue to evolve Project Athena’s testing to reflect trends in how people use their laptops, what they expect and prioritize, and what they don’t even know they need yet.