7 TRANSFORMATION AND QUANTIZATION

While mode selection and local motion compensation are based on the macroblock structure, the transformation and quantization is based on 8*8 blocks.

Blocks are transformed with a 2-dimensional DCT as explained in Appendix A. Each block of 8*8 pixels thus results in a block of 8*8 transform coefficients. The DCT coefficients are quantized as described in sections 7.1 and 7.2.

7.1 Quantization of Intra Macroblocks

Intra frame DCT coefficients are quantized with selected quantizers without a dead-zone.

7.1.1 DC Coefficients

The quantizer step-size for the DC coefficient of the luminance and chrominance components is 8, 4, 2 and 1. Thus, the quantized DC value, QDC, is calculated as:

QDC	= dc // 8
QDC(9bit)	= dc // 4
QDC(10bit)	= dc // 2

where "dc" is the 11-bit unquantized value from the DCT.

7.1.2 AC Coefficients

AC coefficients ac(i,j) are first quantised by individual quantisation factors,

ac~(i,j) = (16 * ac(i,j)) //wI(i,j)

where wI(i,j) is the (i,j)th element of the Intra quantizer matrix given in figure 7.1. ac~(i,j) is limited to the range [-2048, 2047].

The step-size for quantizing the scaled DCT coefficients, ac~(i,j), is derived from the quantization parameter, quantizer_scale. A quantizer_scale is calculated for each macroblock by the algorithm defined in Test Model Section 10 and is stored in the bitstream in the slice header and, optionally, in any macroblock.

The quantized level QAC(i,j) is given by:

QAC(i,j) = [ac~(i,j) + sign(ac~(i,j))*((p * quantizer_scale) // q)] / (2*quantizer_scale)

If the tcoef_escape_format flag is set to 0, QAC (i,j) is limited to the range [-255..255].

If the tcoef_escape_format flag is set to 1, QAC (i,j) is limited to the range [-2047 .. 2047].

For this TM p=3, and q = 4.

7.2 Quantization Non Intra Macroblocks

Non-intra macroblocks in Predicted and Interpolated pictures are quantized with a uniform quantizer that has a dead-zone about zero. A non-intra quantizer matrix, given in figure 7.2, is used.

The step-size for quantizing both the scaled DC and AC coefficients is derived from the quantization parameter, quantizer_scale. quantizer_scale is calculated for each macroblock by the algorithm defined in Section 10. The following formulae describe the quantization process. Note that INTRA type macroblocks in predicted and interpolated pictures are quantized in exactly the same manner as macroblocks in Intra-pictures (section 7.1) and not as described in this section.

ac~(i,j) = (16 * ac(i,j)) // wN(i,j)

where:

wN(i,j) is the non-intra quantizer matrix given in figure 7.2

QAC(i,j) = ac~(i,j) / (2*quantizer_scale)

If MPEG1 syntax applies, QAC (i,j) is limited to the range [-255..255].

If MPEG2 syntax applies, no clipping is necessary.

7.3 Dequantization

An encoder is free to perform quantization in any way it wants (i.e. "guess", use a mirror of the decoder, or use a fuzzy quantum neural net quadratic maximum a posteri prediction estimator with entropy constrained trellis coded feedback). However, the decoder is deterministic and the rules in IS 13818-2 section 7.4 must be followed.