How to Optimize AI Models: Accuracy, Speed, Cost & Size

“Optimize an AI model” means four different things, and most guides blur them together. A data scientist means raise accuracy. An infrastructure engineer means make it faster and cheaper to run. A mobile developer means make it small enough to fit on a phone. A product team building on a language model means get better answers for fewer tokens. The techniques overlap only a little, and using the wrong one wastes weeks.

This guide separates the four goals, gives you an order of operations that saves time, and shows each major technique with real numbers and working code: tuning, pruning, quantization, distillation, batching, fine-tuning with LoRA and QLoRA, and more.

Step 0: What Are You Optimizing?

You cannot maximise everything at once. Quality, speed and cost pull against each other, so name your target before you touch the model.

The “Cheapest First” Ladder

Optimization techniques differ hugely in effort and risk. Work from the cheapest rung up, and stop when you hit your target:

The ladder exists because people jump to rung 5 (a glamorous technique like quantization) when rung 2 (a data problem) is the real cause. The code example below shows how often that happens.

Measure First: Find the Real Bottleneck

Optimizing the wrong part of a system produces no visible gain. The math is unforgiving. If a step takes a fraction p of total time and you make it infinitely fast, the best possible overall speed-up is 1 ÷ (1 − p) (a result known as Amdahl’s law):

The practical rule: profile your pipeline end to end before you optimise anything. In many production systems the slow part is not the model at all. It is data loading, retrieval, network calls, tokenisation or waiting for other services. Fix the biggest slice first.

Goal 1: Improve Accuracy

Diagnose before you treat

The gap between training and test performance tells you what is wrong:

Example 1: Fix the data before you tune

The code below trains a logistic regression on a standard medical dataset. Look at which change moves the score and which barely does.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

X, y = load_breast_cancer(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

raw    = LogisticRegression(max_iter=5000).fit(Xtr, ytr)
scaled = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)).fit(Xtr, ytr)
print("raw features    test acc:", round(raw.score(Xte, yte), 3))      # 0.937
print("scaled features test acc:", round(scaled.score(Xte, yte), 3))   # 0.958

grid = GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)),
                    {"logisticregression__C": [0.001, 0.01, 0.1, 1, 10, 100]}, cv=5).fit(Xtr, ytr)
print("tuned C =", grid.best_params_["logisticregression__C"],
      "cv:", round(grid.best_score_, 3), "test:", round(grid.score(Xte, yte), 3))   # C=0.1  cv 0.979  test 0.958
print("default C=1 cv:", round(cross_val_score(
      make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)), Xtr, ytr, cv=5).mean(), 3))  # 0.974

What this shows: a one-line data fix delivered more than a whole hyperparameter search, and the search improved the cross-validation score by half a point without improving the held-out test result at all. That is normal. Tuning is the last mile, and it can overfit its own validation data if you push it too far. Always judge a tuned model on data it never saw during the search.

Tuning methods in one table

The most important setting for neural networks is usually the learning rate, followed by batch size and regularisation strength. Tune those first, and always use cross-validation or a separate validation set, never the final test set.

Goal 2: Make the Model Smaller

Compression trades a little quality for large savings in memory and compute. Three techniques dominate, and they combine well.

1. Quantization: fewer bits per number

Neural network weights are normally stored as 32-bit or 16-bit numbers. Quantization stores them with fewer bits, such as 8-bit integers (INT8) or 4-bit values. The memory arithmetic is simple: 16-bit to 8-bit halves the size, and 16-bit to 4-bit cuts it to a quarter.

Two approaches exist. Post-training quantization converts an already trained model, which is quick and needs little data. Quantization-aware training simulates low precision during training, which is more work but recovers more quality at very low bit-widths.

Example 2: Quantize a model and measure the damage

import numpy as np

coef = scaled[-1].coef_.ravel().astype(np.float32)      # trained weights (30 numbers)
b = scaled[-1].intercept_

def quantize(w, bits):
    levels = 2**(bits - 1) - 1
    scale = np.abs(w).max() / levels
    return np.round(w / scale).astype(np.int8), scale

Xs = scaled[0].transform(Xte)
base = (Xs @ coef + b > 0).astype(int)
for bits in (8, 4):
    q, s = quantize(coef, bits)
    deq = q.astype(np.float32) * s
    pred = (Xs @ deq + b > 0).astype(int)
    print(bits, "bit | max weight error", round(np.abs(coef - deq).max(), 4),
          "| agrees with float32 on", f"{np.mean(pred == base):.1%}", "of predictions")

# 8 bit | max weight error 0.0048 | agrees with float32 on 100.0% of predictions
# 4 bit | max weight error 0.0855 | agrees with float32 on 100.0% of predictions

What this shows: dropping from 32-bit to 8-bit made the weights four times smaller with a tiny error, and 4-bit made them eight times smaller with a larger error, yet predictions did not change on this small model. Large language models have billions of weights and sensitive outlier values, so they lose more quality when quantized aggressively. The lesson is the method, not the result: quantize, then re-run your full evaluation set. The authors of QLoRA reported that 4-bit NormalFloat base weights preserved full 16-bit fine-tuning performance on the tasks they tested, but your task is the one that counts.

2. Pruning: remove what the model does not need

Unstructured pruning zeroes out individual weights, which shrinks the file but needs special software or hardware to actually run faster. Structured pruning removes whole neurons, channels or layers, which standard hardware can speed up directly. Pruned models usually lose some accuracy, which a short round of retraining can recover.

Example 3: Pruning can even improve accuracy

Decision trees have a built-in pruning setting called ccp_alpha. The unpruned tree below memorises its training data, and pruning makes it smaller and better at new data:

What this shows: trimming the tree from 35 nodes to 11 cut its size by about two thirds and raised test accuracy from 90.2% to 92.3%, because the removed branches had only memorised noise. Prune too far (3 to 7 nodes) and accuracy falls again. For neural networks the effect is gentler, but the same logic holds: moderate pruning often costs little, and heavy pruning needs retraining.

3. Knowledge distillation: a small student learns from a large teacher

Instead of shrinking the big model, you train a smaller one to imitate it. The classic result is DistilBERT, which was 40% smaller than BERT, 60% faster at inference and retained 97% of its language-understanding performance on the GLUE benchmark. Distillation also works for language models and image models, and it is how many fast production models are produced from larger ones.

Compression cheat sheet

Goal 3: Make It Faster and Cheaper to Run

Serving optimization changes how the model is run, not the model itself, so quality stays the same or nearly the same. This is where the biggest cost savings often hide.

Batching: do more work per trip

Hardware, especially GPUs, is built to process many items at once. Running requests one at a time wastes most of the chip. Batching groups requests together.

import time, numpy as np
rng = np.random.default_rng(0)
W = rng.standard_normal((512, 512)).astype(np.float32)
data = rng.standard_normal((2000, 512)).astype(np.float32)

t0 = time.perf_counter(); out1 = [x @ W for x in data]; one_by_one = time.perf_counter() - t0
t0 = time.perf_counter(); out2 = data @ W;               batched = time.perf_counter() - t0
print(f"one-by-one: {one_by_one*1000:.1f} ms   batched: {batched*1000:.1f} ms   speedup: {one_by_one/batched:.0f}x")
print("same results:", np.allclose(np.array(out1), out2, atol=1e-3))

# one-by-one: 46.9 ms   batched: 14.3 ms   speedup: 3x
# same results: True

What this shows: the same calculation, with identical results, ran about 3 times faster when batched, even on an ordinary CPU. Your exact timing will differ, and on a GPU the effect is usually larger. The catch is latency: waiting to fill a batch delays each request, so real systems balance batch size against response time.

The LLM serving toolbox

Large language models add their own bottlenecks, because they generate text one token at a time and keep a growing memory of the conversation (the KV cache). These techniques address them:

How to read the gains: published numbers come from different hardware, models and baselines, so treat them as ranges, not promises. Some write-ups report that stacking quantization, memory-efficient caching and speculative decoding can cut inference cost by 5 to 10 times in total. Speculative decoding only helps when generation is memory-bound, and acceptance of the draft’s tokens should stay high (60% or more is a common rule of thumb), or the extra work cancels the benefit. Measure on your own traffic.

Runtimes and compilers

Exporting a model to an optimised runtime such as ONNX Runtime, TensorRT or a framework compiler (for example PyTorch’s torch.compile) fuses operations and picks faster kernels for your hardware, often for a one-time engineering cost. On mobile and edge devices, toolchains such as TensorFlow Lite, PyTorch Mobile and Qualcomm AI Hub play the same role.

Goal 4: Optimize an LLM Application

If you build on a hosted or open language model, most of your optimization happens above the model. The ladder here is different, and again you climb it cheapest first:

The rule of thumb: fine-tuning changes how a model behaves, and retrieval changes what it knows. Use prompts and retrieval before you fine-tune, and evaluate each change on the same test set so you know what actually helped.

LoRA and QLoRA: fine-tuning on a budget

Full fine-tuning updates every weight and needs memory for gradients and optimizer state on top of the weights. LoRA freezes the original model and trains small add-on matrices instead. In the original paper’s GPT-3 175B experiments, LoRA cut trainable parameters by about 10,000 times and GPU memory needs by about 3 times compared with full fine-tuning, while matching or beating its quality on the tasks tested. QLoRA goes further by storing the frozen base model in 4-bit precision, which made it possible to fine-tune a 65-billion-parameter model on a single 48 GB GPU while preserving 16-bit fine-tuning performance on the authors’ tests.

Token optimisation: the free lunch

• Shorten prompts: remove boilerplate and send only the context that matters.
• Cap output length: ask for concise answers and set maximum output limits.
• Cache: reuse answers and prompt prefixes for repeated requests.
• Route: use small models for classification, extraction and simple replies.
• Use structured output: constrained formats cut retries and parsing failures.

Memory and Cost Math

You can estimate whether a model fits on your hardware with simple arithmetic. The memory for weights is parameters times bytes per parameter. Training adds gradients and optimizer state, which for the common Adam optimizer in mixed precision comes to roughly 16 bytes per parameter in total.

def gb(params_billions, bytes_per_param):
    return params_billions * bytes_per_param      # billions of params x bytes = GB

for name, p in [("7B", 7), ("70B", 70)]:
    print(f"{name}: fp16 weights {gb(p, 2):6.1f} GB | int8 {gb(p, 1):6.1f} | "
          f"4-bit {gb(p, 0.5):6.1f} | full fine-tune (Adam, ~16 B/param) {gb(p, 16):7.1f}")

# 7B:  fp16 weights   14.0 GB | int8    7.0 | 4-bit    3.5 | full fine-tune (Adam, ~16 B/param)   112.0
# 70B: fp16 weights  140.0 GB | int8   70.0 | 4-bit   35.0 | full fine-tune (Adam, ~16 B/param)  1120.0

What this shows: a 7-billion-parameter model needs 14 GB just to hold its weights in 16-bit form, but about 112 GB to fine-tune fully, because gradients and optimizer state dwarf the weights themselves. That is why LoRA, which needs gradients only for tiny adapters, and QLoRA, which also shrinks the frozen base to 4 bits, turn an impossible job into a single-GPU job. These figures cover weights and training state only. Activations and the KV cache add more during use, so leave headroom.

Which Technique Should You Use?

How to Benchmark Without Fooling Yourself

Most failed optimizations are measurement mistakes. Follow these rules every time:

Optimizing for Your Deployment Target

Common Mistakes

Optimization Checklist

Common Myths About Optimizing AI Models

Mini Glossary

• Latency: how long one request takes. Throughput: how many requests per second the system handles.
• Overfitting: memorising training data so performance on new data is worse.
• Hyperparameter: a setting chosen before training, such as learning rate.
• Quantization: storing a model’s numbers with fewer bits.
• Pruning: removing weights, neurons or layers a model does not need.
• Distillation: training a small model to imitate a large one.
• LoRA / QLoRA: memory-efficient fine-tuning that trains small adapter matrices (QLoRA on a 4-bit base model).
• KV cache: the stored attention data a language model keeps for the conversation so far.
• Speculative decoding: a small model drafts tokens and the large model verifies them in one pass.
• Continuous batching: letting new requests join a running batch as others finish.
• RAG: retrieval-augmented generation; giving a model relevant documents to answer from.

Read More: How to Measure AI Performance?

Frequently Asked Questions About Optimizing AI Models

How do you optimize an AI model?

Name the goal (accuracy, size, speed or cost), measure a baseline on held-out data, find the bottleneck, then apply the cheapest fix first: better data, right-sizing, tuning, compression, then efficient serving. Re-test quality after every change.

What are the main techniques for optimizing AI models?

Data cleaning and feature work, hyperparameter tuning and regularisation for accuracy; quantization, pruning and distillation for size; batching, caching, optimised runtimes and speculative decoding for speed and cost; and prompts, retrieval, fine-tuning and routing for language-model applications.

What is the difference between quantization and pruning?

Quantization stores each weight with fewer bits. Pruning removes weights, neurons or layers entirely. They can be combined, and both usually need a quality re-check afterwards.

What is knowledge distillation?

Training a small student model to reproduce the outputs of a large teacher model. DistilBERT is the classic example: 40% smaller, 60% faster and 97% of BERT’s language-understanding performance.

Should I fine-tune or use RAG?

Use retrieval (RAG) when the model needs facts, especially private or changing ones. Use fine-tuning when you need consistent behaviour, tone or format that prompts cannot achieve. Many systems use both.

What are LoRA and QLoRA?

Methods for fine-tuning large models cheaply. LoRA trains small adapter matrices and freezes the base model, cutting trainable parameters by about 10,000 times in the original GPT-3 experiments. QLoRA adds a 4-bit base model, letting a 65-billion-parameter model be fine-tuned on one 48 GB GPU.

How do I make an LLM faster and cheaper to run?

Use a quantized or smaller model, a modern serving engine with continuous batching and efficient KV-cache management, speculative decoding where it helps, prompt and prefix caching, and routing so easy requests use cheaper models.

How much memory do I need to fine-tune a model?

Full fine-tuning with Adam needs roughly 16 bytes per parameter, so about 112 GB for a 7-billion-parameter model. LoRA and QLoRA cut this drastically by training only small adapters on a frozen (and, for QLoRA, 4-bit) base model.

Does optimizing a model reduce its accuracy?

Not always. Tuning, data fixes and moderate pruning can raise accuracy, while compression can lower it slightly. The only way to know is to evaluate the optimized model on the same held-out test set as the original.

How do I optimize an AI model for mobile?

Choose a small architecture or distil a larger one, quantize aggressively, convert to a device runtime such as TensorFlow Lite, and test speed, memory and battery on the actual device.

How do AI agents for business differ from traditional AI models?

While a standard AI model acts as a static cognitive engine that answers isolated prompts, an AI agent for business is an autonomous system designed to execute complex, multi-step workflows. It uses tools (such as databases, APIs, and web search), maintains an operational state, and performs end-to-end tasks—like automated customer support resolution, market research, or inventory tracking—without needing constant human intervention.

How do model optimization and agentic workflows relate to the path toward Superintelligence?

Optimizing foundation models and building advanced agentic systems are critical stepping stones on the roadmap toward Superintelligence. As models become faster and more efficient through quantization, pruning, and distillation, AI agents gain the computational bandwidth to reason, self-correct, and execute long-horizon tasks autonomously at scale, bringing systems closer to general problem-solving capabilities.

Why is optimizing an AI agent for business crucial for enterprise scalability?

Unlike single-turn API prompts, an AI agent for business typically runs multiple reasoning loops, tool calls, and verification steps per user request. This multi-step execution can rapidly inflate compute costs and latency. Optimizing the underlying models—via quantization, prompt caching, and efficient serving runtimes—ensures that enterprise automation remains economically viable and lightning-fast.

Key Takeaways

• “Optimize” means four goals: accuracy, size, speed and cost, plus language-model application quality. Pick one first.
• Climb the cheapest-first ladder: measure, fix the data, right-size, tune, compress, serve efficiently, monitor.
• Profile before optimizing; a step that is 10% of runtime can give at most a 1.1 times overall speed-up.
• Data fixes usually beat tuning: in our example, scaling added 2.1 points and tuning added none on the test set.
• Quantization cuts memory 2 to 4 times, pruning can improve generalisation, and distillation can keep about 97% of quality in a much smaller model.
• For language models, use prompts and retrieval before fine-tuning, and use LoRA or QLoRA when you do fine-tune.
• Batching, continuous batching, KV-cache management, caching and routing deliver most serving savings.
• Benchmark honestly: held-out data, one change at a time, real hardware, the slowest 5% of requests, and a quality re-test after every step.

We refresh the dated snapshot tables as new techniques and measurements appear, while the principles and the math stay the same. Bookmark this page, and follow the AICopse AI Updates section and the AICopse homepage for daily AI coverage.