Skip to content Skip to footer
NStarX Inc.
Get in Touch
NStarX Inc.
Let's Talk
Close
Case Study

AI PC Evaluation and Benchmarking for Generative AI Workloads

January 29, 2025

Business Challenge

Situation

A global enterprise aimed to evaluate the performance of endpoint devices for deploying Generative AI workloads, focusing on advanced use cases like Retrieval-Augmented Generation (RAG). The objective was to assess the feasibility of utilizing state-of-the-art hardware, such as Intel NPU-enabled laptops, to support enterprise AI applications.

Challenges

Performance Limitations

  • Ensuring reliable AI processing for computationally intensive workloads like querying large language models (LLMs).
  • Addressing latency and throughput issues for real-time responses.

Hardware Compatibility

  • Adapting the AI application to various hardware architectures, including Meteor Lake, Alder Lake-U, and legacy laptops.

Optimization Needs

  • Balancing performance and accuracy through techniques like model quantization.

Autonomous Operation

  • Developing a secure, air-gapped solution where all AI processing occurs locally on the endpoint device.

Solution

To address these challenges, a state-of-the-art LLM-based RAG-driven chatbot application was developed:

Technical Design

  • Built using the Llama3 model with 7B parameters, optimized by quantizing to int4 for efficient processing.
  • The chatbot supported both text-based and audio-based interactions, delivering accurate and contextually relevant responses.

Deployment Tested

  • across Meteor Lake-based AI PCs, Alder Lake-U devices, and legacy laptops to evaluate performance across hardware types.
  • Key performance metrics like inference rate, memory usage, and CPU/GPU/NPU utilization were monitored.

Secure Processing

  • All AI computations were performed locally, ensuring compliance with stringent security and privacy requirements.

Optimization Techniques

  • Leveraged quantization and hardware-specific optimizations to enhance the AI application's performance while minimizing resource utilization.

Technology & Tools

Hardware:
Intel Meteor Lake (32GB, Intel Core Ultra 7 155U), Alder Lake-U (64GB, Intel Core i7), and legacy Intel Core i5 laptops.

Framework:
Intel OpenVINO for model training, hosting, and querying.

Model:
Llama3 with 7B parameters, quantized to int4 for optimized performance.

Workload:
RAG-driven chatbot trained on domain-specific datasets.

Business Outcomes

Performance Insights

  • Meteor Lake-based AI PCs demonstrated superior performance in inference rates and reduced latency compared to legacy devices, highlighting their potential for enterprise AI workloads.

Scalability

  • Optimized LLM deployment on endpoint devices showcased the potential for scaling
  • Generative AI applications without heavy infrastructure dependencies.

Security

  • The air-gapped deployment ensured data privacy and compliance, a critical requirement for enterprise use cases.

Actionable Recommendations

  • Insights were provided to enhance performance further, including porting the application to additional hardware stacks (e.g., Apple, Qualcomm, AMD) and expanding support for sophisticated AI workloads.
Privacy Overview
NStarX Logo

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Necessary

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.