Joint Optimization of Transform and Quantization for High Efficiency Video Coding

In high efficiency video coding (HEVC), transformation and quantization are separately performed to eliminate the perceptual redundancy of visual signals. However, a uniform quantizer can inevitably degrade the compression efficiency of fixed transform matrices due to varying space-frequency characteristics of video content. This paper introduces a joint optimization of transform and quantization approach for video coding. First, we compute a content dependent transform from the reconstructed reference by a fast Karhunen-Loéve transform (KLT). Second, using a template-based rate regularization, we jointly optimize transform and quantization (JOTQ) as a rate constrained optimization problem and obtain a feasible solution to improve coding performance. Finally, we design fast algorithms and early terminations to reduce the computational complexity of JOTQ. The experimental results show that JOTQ outperforms several previous methods by providing Bjontegaard Delta rate reductions of 4.11% and 3.38% on average under the low-delay and random-access configuration, respectively.