My Laptop Has an Unused NPU. I Want to Finally Find Out Whether It Can Be Useful

My Lenovo Yoga Slim 7x has another processor alongside the usual CPU and GPU: an NPU (Neural Processing Unit), designed to run neural network computations in an energy-efficient way. In everyday use, apart from things like blurring the background during calls or indexing files in the background, it mostly sits idle. I decided to check what else I could do with it, how fast it is, and whether it is worth using in practice.

I started by checking which language models I could run on it, which tools I needed, and how much work the setup took. I measured generation speed, response latency, the effects of context settings and input length, and behavior on battery. I did not obtain reliable RAM measurements or evaluate response quality.

I hope a local LLM will be useful when the connection to the cloud is poor or unavailable, for example on a plane or a train. It might be enough to help with text, code, or notes stored on the laptop. It is also useful to know which tasks I can handle without sending a prompt to an external model. An NPU could be particularly useful here: it was designed to run neural networks using less energy than a CPU or GPU, and ASUS explicitly connects NPUs with lower power consumption and longer battery life (ASUS). I compared speed and collected preliminary power readings, but did not measure battery runtime.

I did not assume that a local model would replace the cloud, or that the NPU had to beat the CPU or GPU. The experiment explored when it is worth using the NPU, which execution path to choose, and where its practical usefulness ends. Finding scenarios where the CPU or GPU was more practical was a useful result too.

Another motivation is simply to learn more about the technology and experiment with model settings.

The starting point was Qwen3-4B-Instruct-2507. I tested F16, Q8_0, Q4_K_M, and Q4_0 GGUF variants and a prepared W4A16 QNN package, then compared Qwen3 sizes from 1.7B to 32B on the CPU. I also varied context window capacities and actual input lengths. All comparisons ran on native Windows ARM64 on this laptop.

Key results:

• Short input, AC power: the CPU with Q4_0 gives about 33 tokens/s - the fastest.
• Battery + Power Saving: the NPU (qairt) maintained about 20 tokens/s, making it my starting point for responsive use in this mode.
• Long input (6180 tokens): about 33 s on the NPU versus about 86 s on the CPU.
• Model size: 8B and 14B offered usable interactive speeds; I found 32B (about 4.8 tokens/s) too slow for comfortable interactive chat in this test.
• The NPU is not automatically the best choice - it depends on the scenario.

The power figures in the chart below are approximate net SYS readings from separate single-run measurements, with responses of different lengths and content. Counter coverage and measurement error have not been established. These are not measurements of battery runtime or energy for an identical task.

Longer input can change which device is faster. With 420 input tokens, the CPU finished generation sooner: 6.69 s versus 9.31 s on the NPU. With 6180 input tokens, the NPU took 32.96 s and the CPU 86.03 s - the NPU was about 2.6× faster in this test.

1. Background

1.1. Hardware

I’m testing a Yoga Slim 7x with a Snapdragon X Elite X1E-78-100 and 32 GB of RAM. There are three compute devices within this single SoC that can run model computations.

The laptop has a Qualcomm Snapdragon X Elite with three compute units:

• CPU - 12 Qualcomm Oryon cores, split into three clusters of four, running at up to 3.4 GHz under multicore workloads.
• GPU - integrated Qualcomm Adreno X1-85, using shared RAM, with OpenCL support.
• NPU - Qualcomm Hexagon, rated at up to 45 TOPS (trillion operations per second).

TFLOPS and TOPS don’t translate directly into tokens/s. Generation speed also depends on the model, data transfers from memory, and the compute implementation. It needs to be measured for a specific configuration.

x → MatMul with W → Add b → Relu → y

One Q4_0 block:
[ scale: 2 bytes ] + [ 32 codes of 4 bits each: 16 bytes ]

Total: 18 bytes for 32 weights.
This averages 4.5 bits per model parameter.

1.5. Reading the Timing, Context, and Memory Results

• Prefill: processing the input.
• TTFT: time from the start of a request to the first response token.
• Decode: the rate of generating subsequent tokens, in tokens/s.

Reported TTFT and generation times do not include loading the model. A shorter TTFT does not necessarily mean the full response finishes sooner.

Window capacity is not input length. The input, history, chat markers, and response share the context budget. The KV cache stores part of the computations for retained tokens, but a longer history still increases memory and attention costs. This is why I vary window capacity and actual prompt length separately.

Model file size is not the same as RAM use. There are also caches, activations, buffers, and memory used by the system. I did not obtain reliable RAM measurements.

2. What Actually Ran

First, I checked whether each environment could initialize the model and generate a response at all. Only stable, working configurations made it into the final performance measurements.

2.1. Environment Summary

Setup and Problems

Standalone llama.cpp

The ARM64 build for the CPU compiled successfully and runs reliably. The version with the ggml-hexagon backend detects the NPU, but loading the model fails when it tries to open a session. I have not established the cause. This does not show that the hardware cannot run the model: the same compute path worked through the llama.cpp libraries included with GenieX.

GenieX with the llama_cpp Plugin (GGUF)

I registered a local GGUF file and tested NPU initialization. The performance measurements covered CPU, GPU, NPU, and hybrid modes. In hybrid mode, the actual distribution of work between the SoC’s compute units remains a black box from the user’s perspective.

GenieX with the qairt Plugin (QNN Package)

I used a prepared QNN W4A16 package for Snapdragon X Elite chips, compatible with the qairt plugin.

2.2. What I Did Not Measure

I considered ORT as an alternative, but testing it would require preparing or obtaining a compatible model, configuring generation, and verifying the selected integration. That would be a separate stage of work requiring more time. I therefore limited the scope to working GenieX configurations. The lack of ORT results is a scope decision, not evidence that this path does not work.

3. How I Measure and What I Compare

I tested GenieX v0.10.0 with QAIRT 2.50 and its bundled llama.cpp version.

I ran the speed measurements natively on Windows ARM64 through GenieX. For the speed and quantization tables, each completed cell consists of one warm-up run and five measured runs. I report the median. Window capacity and input length tests have three measured runs each. Settings are a limit of 128 new tokens, non-thinking mode, temperature 0, and seed 42. In the baseline 4B comparison, llama_cpp used a 512-token window, while qairt used the package configuration. The separate input length series use a fixed 16,384-token window.

Each run is the first request in a new process. A warm-up run may warm the system and file cache, but not the runtime instance used by the next run. Reported generation times exclude loading the model. The same prompt text produces 48 input tokens in llama_cpp and 61 in qairt. The two paths use different artifacts and configurations, so the TTFT comparison does not isolate the effect of hardware.

I set the power states manually: AC power, battery, and battery with Power Saving.

4. Results: Speed, Energy, and Input Length

The charts show results from recorded runs. Error bars on the speed charts show the minimum and maximum, not a confidence interval. AC means connected to the charger, and Power Saving means the energy-saving mode.

4.1. Speed: 4B in Three Power States

Medians from five runs after one warm-up run. The exception is the CPU with Power Saving, where the warm-up run timed out. The response limit is 128 tokens, with thinking disabled. The llama_cpp window is 512 tokens; the qairt window comes from the package.

qairt had the shortest TTFT: medians of 89–98 ms depending on the power state. The CPU achieved 150–158 ms, the NPU through ggml-hexagon 158–187 ms, hybrid 207–220 ms, and the GPU 278–291 ms. These are results for complete execution paths, not the devices alone: qairt processed 61 input tokens, while llama_cpp processed 48.

4.2. Average Power: How Many Watts Did the Measurement Show?

To make the comparison easier to read, I show average net power in watts (W). Watts describe how quickly energy is being used, not how much energy the entire response consumed. Lower power does not always mean less energy per task if the model has to run much longer.

These are estimates from the SYS channel after subtracting a separately measured idle baseline. They are not the laptop’s total wall power or a measurement of the NPU alone. I calculated the average from the recorded energy and elapsed run time, including model loading. Historical readings from the two series required different unit conversions. The exact counter coverage and measurement error have not been established.

In simple terms: in a separate AC-powered run, the readings were about 44 W for the CPU, 18 W for the GPU, and 9 W for qairt on the NPU. On battery with Power Saving, qairt showed about 4 W and the GPU about 8 W. This looks promising for the NPU, but it is not a battery life prediction.

There is only one run per configuration, and the responses differ in both length and content. I therefore do not treat these numbers as the cost of performing an identical task. The CPU timeout is a separate observation, not a completed test. The battery series changes both the power source and the operating mode, so it does not isolate the effect of Power Saving alone.

4.3. What Would I Choose on Battery and with Power Saving?

Unplugging the charger alone did not change speed much in this short test: decode differed by less than 1% for the CPU and both NPU paths, about 4.6% for the GPU, and 4.1% for hybrid.

With Power Saving, I would choose the NPU for its tested speed and responsiveness. qairt maintained about 20.4 tokens/s, and ggml-hexagon 16.7 tokens/s. The GPU dropped to 13.6 tokens/s, hybrid to 11.3 tokens/s, and the CPU did not finish the warm-up within the 15-minute limit. During that run, I observed activity on 8 of the 12 threads, but did not establish the cause of the delay.

The NPU was therefore my preferred tested path for responsive use in this mode, but not the only one that worked: the GPU still generated text. The separate lower approximate SYS power reading for qairt is preliminary supporting evidence, not proof that it provides the longest battery runtime.

4.4. Quantization: 4B on CPU and NPU through ggml-hexagon

AC power, a short prompt, medians from five runs. The chart shows decode: the rate of generating subsequent tokens.

The separate qairt QNN W4A16 package achieved 21.3 tokens/s. I do not include it in the same GGUF quantization series.

On the CPU, Q4_0 was about 3.2 times faster than F16 in decode and more than 12 times faster in prefill. The NPU had slower decode for all these GGUF variants, but smaller prefill differences between quantization schemes. This is a result for this model and build, not a general ranking of the schemes. This series measured speed, not response quality.

4.5. Parameter Count: From 1.7B to 32B on the CPU

AC power, Q4_K_M, non-thinking, five measured runs each.

• These are original Qwen3 models with switchable thinking/non-thinking modes, tested here with thinking disabled. The 4B-2507 model is a different checkpoint, so it does not belong directly in this series.

Which of These Are Practical?

• 1.7B: the most speed headroom. A useful starting point for simple, narrow tasks, but check whether the quality is enough.
• 8B: about 18 tokens/s with a short input is a reasonable starting point for interactive work.
• 14B: about 10.5 tokens/s. Slower, but still a reasonable option if the larger model gives a better result for a particular task.
• 32B: about 4.8 tokens/s. Decoding 128 tokens alone takes roughly 27 seconds: too slow for comfortable interactive chat in this test. It could still be useful for batch tasks if waiting is acceptable, but its quality for those tasks needs assessment; I did not measure it.

32B is the largest model for which I obtained a result in this series, not an established limit of this laptop. These runs use short inputs and a 512-token window. Speed does not demonstrate quality, and a longer document could change responsiveness considerably.

4.6. Window Capacity and Actual Input Length

Changing Window Capacity with a Short Prompt (CPU, AC)

I varied the window capacity while keeping 52 input tokens and 128 response tokens. The history therefore grew to about 180 positions, not 65k or 262k. Below are median TTFT values from three runs per cell.

TTFT increased at larger limits, but not monotonically. I did not establish the cause or a practical document length limit: this test did not fill the window.

Longer Inputs with a Fixed Window (4B Q4_0, CPU and NPU)

On AC power with the maximum performance profile, I kept the window at 16,384 tokens and increased the length of a synthetic input made of repeated paragraphs. I ran the CPU series first, then the NPU through GenieX llama_cpp, meaning ggml-hexagon, not qairt. I used the same registered Qwen3-4B-Instruct-2507 Q4_0, identical prompts, temperature 0, and three measured runs per cell after one warm-up run. Each run used a new process, and each response had exactly 128 reported tokens and stopped at the length limit.

Total generation is the median of the engine-reported total generation time, excluding model loading, not the sum of medians for other metrics. CPU prefill was 211.2, 182.1, and 94.9 tokens/s respectively; NPU prefill was 640.9, 577.1, and 376.4 tokens/s.

With longer inputs, the NPU can be faster than the CPU. For 6180 input tokens and 128 response tokens, generation took about 33 seconds on the NPU versus 86 seconds on the CPU. With the shortest input, the CPU finished first. The choice therefore also depends on input and response length.

These are two separate findings: I measured the speed advantage for longer inputs through ggml-hexagon, and lower power in another series through qairt. I did not measure power or energy for this longer-input test.

Between the shortest and longest inputs, CPU decode dropped by about 77% and NPU decode by 48%. I do not attribute the drop to a single mechanism: the input is synthetic, the series ran from shortest to longest, and I did not record temperature or clock speeds. I did not measure energy or document task quality here.

5. What to Choose for a Particular Use Case

5.1. Practical Conclusions from These Tests

I would not pick one configuration for everything. First, I would decide whether response speed, battery use, document length, or the capabilities of a larger model matter most. The suggestions below are starting points for your own tasks, not a model quality ranking or a guarantee of how every application will behave.

5.2. My Takeaways

The NPU turned out to be useful, and I will keep experimenting with it. It is not automatically the best choice just because it was designed for neural network computations:

• Short input and AC power: the CPU with Q4_0 gave the highest decode throughput for the tested 4B model. I don’t need to force the NPU into the workflow for a quick response.
• Power Saving: the NPU maintained responsiveness better than the GPU and hybrid mode. That tested speed makes qairt my first candidate in this mode; the separate lower SYS power reading remains preliminary, rather than evidence of longer battery life.
• Longer input: ggml-hexagon could beat the CPU on total response time. A short-question benchmark is not enough to choose a configuration for documents.
• Model size: 8B and 14B look reasonable in terms of speed. 32B (about 4.8 tokens/s) was too slow for comfortable interactive chat in this test, but could still be useful for batch tasks if waiting is acceptable. Whether more parameters improve the result needs a task-specific quality assessment, which I did not perform.

The next step is to check whether the selected models actually handle real tasks well: editing text, summarizing notes, extracting information from documents, or helping with code.

I also want to see how to turn these working configurations into everyday use, for example in LM Studio, Ollama, or AnythingLLM.

LLMs are not the only possible use for the NPU. Audio transcription and image processing are other directions to explore, with different models and requirements.

P.S. A note on the language: this time I decided to try something new and wrote the article in Polish, my native language. I can put the text together faster and make fewer mistakes that way. LLMs seem to be very good at translation, so I used one to translate it into English. I’m curious to see how well this works.