Quantization: how amplitude becomes a code

← The Nyquist frequency

Two limits of a digital record

Until now, we selected the sampling instants and used ideal numbers for the sample values. A real analog-to-digital converter (ADC) also limits the precision in amplitude:

1. Sampling selects the time instants $t_n=n/F_s$.
2. Quantization replaces the measured value with one level from a finite set of levels and gives the related digital code.

Resolution is the number of bits in the code. More bits let you tell apart amplitude values that are nearer to each other, but they do not correct the aliasing from the previous lessons. Here we change the resolution and keep the sample rate the same.

A converter model with positive and negative levels

We measure the signal in arbitrary units: the bottom of the scale is $-1$, and the top is $+1$. For a resolution of $N$ bits, there are $L=2^N$ codes. In the selected model, the step between levels is

With $N=3$, we get eight signed codes and a step of $0.25$:

This model has no $+1$ level. This comes from the range of signed integers that we selected: from $-4$ to $+3$. This convention agrees with two’s complement, which we discuss in more detail when we move to fixed-point. A specific ADC can have a different code format.

From value to code and back

For each input value $x[n]$, do three steps:

1. Divide by the step $\Delta$.
2. Round to the nearest integer.
3. If the result is outside the code range, replace it with the nearest limit of the code range.

As a formula:

Here round means rounding, and clip means limiting the range. The script uses np.rint: a value exactly in the middle between two integers is rounded to the nearest even integer.

To show the value of a code on a plot, multiply the code by the step:

For example, with three bits, the input $0.62$ gives $0.62/0.25=2.48$. After rounding, the code is $2$, and the quantized value is $0.50$. The input $0.74$ gives code $3$ and value $0.75$.

What a digital signal contains

The ADC gives codes. In our example, the code sequence contains the integer 2. On the selected scale, it means the value 0.50.

In Python, you can work with both representations:

codes = np.array([2, 3, -2])  # integer codes
step = 0.25
values = codes * step        # [0.50, 0.75, -0.50]

The values array is convenient for plots and for more calculations. Conversion to floating-point numbers does not recover the lost precision: the original values 0.62 and 0.50 could give the same code. To convert normalized values to volts, you also need the input range scale of the device.

Quantization error and range limit

We define the error as the difference between the quantized value and the original value:

If the rounded code is in the valid range, then

If the code had to be limited, this guarantee is lost. For example, for the input $0.90$, rounding gives code $4$, but the maximum code is $3$. We get the value $0.75$ and the error $-0.15$; its magnitude is more than half the step, $0.125$. For the input $1.20$, the error is even larger.

This limit is clipping (the signal is cut off at a level). In our asymmetric model, clipping can start before $+1$: use the valid codes as the reference, not only the nominal limits of the scale.

Quantization error does not always look like random noise. For a periodic signal, it can also be periodic. We examine its spectrum after we learn the DFT.

Lab: resolution and clipping

The script compares models with a resolution of 3 and 8 bits:

uv run labs/01/04-quantization.py --show

The gray line is the original cosine, the blue dots are ideal samples, and the orange dots are the values of the codes. Steps connect the orange dots to make the plot easier to read; the array contains separate samples. The bottom panel shows the error. The script also prints the codes and their values separately.

First compare a few resolutions, then increase the amplitude:

uv run labs/01/04-quantization.py --bits 2 3 4 8 --samples 48 --show
uv run labs/01/04-quantization.py --bits 3 --amplitude 1.2 --show --output-dir /tmp/dsp-clipping

In the second experiment, the peaks hit the limit codes. More bits make the step smaller, but they do not make the selected input scale wider.

What one more bit gives

Each additional bit doubles the number of levels and makes the step two times smaller. When rounding does not need code limiting, the upper limit of the error also becomes two times smaller.

In a real ADC, its own noise and nonlinearity also limit the precision. So the nominal resolution and the actual precision can be different. We keep the quantitative estimates of noise and the effective number of bits (ENOB) for the related part of the course.

Self-check

For the three-bit model, the step is $\Delta=0.25$, and the codes are in the range from $-4$ to $3$. We get the code by rounding $x/\Delta$ to the nearest integer and then limiting the range. The quantized value is $x_q=k\Delta$.

Next lesson: a complex number as a vector — I and Q →