HEIC and AVIF Under the Hood

Underneath the marketing, both modern photo formats rest on the same trick: a single video keyframe sealed inside an MP4-family box. This article opens the box and follows your pixels stage by stage, explaining why each stage beats the 1992 design it replaced.

Two companion articles tell the surrounding story, covering why Apple bet the iPhone camera on HEIC and how AVIF compares in practice against JPEG and WebP. This one is for readers who want the mechanism. You don't need any prior codec knowledge, but we will go all the way down.

The container is an MP4 relative

Start with the wrapper, because it explains the family resemblance. Both HEIC and AVIF use the ISO Base Media File Format (ISO-BMFF), the box-structured container underneath every MP4 video. HEIF is the specification that defines how still images live inside that container. HEIC is HEIF carrying an HEVC-coded image; AVIF is a HEIF-derived specification (MIAF) carrying an AV1-coded image. It is the same shelving system with a different codec on the shelf.

The container stores each image as an "item", with dedicated boxes for the coded image data, EXIF metadata, and color profiles (ICC, or the more compact CICP signaling). Auxiliary images such as alpha transparency planes and depth maps are stored as separate coded images that the main item references, which is how an iPhone portrait photo carries its depth data in the same file. There are boxes for thumbnails, for image sequences and bursts, and for derived items like grids that assemble many coded tiles into one giant picture.

JPEG, by contrast, is a bare stream of segments. Everything beyond the pixels, like EXIF in an APP1 segment or ICC profiles split across APP2, is a convention bolted on after the fact, honored by most software and mangled by some. In HEIF, those are first-class citizens of a real container. That structural difference matters as much as any compression gain.

Block partitioning puts bits where the detail is

Every codec in this story starts by chopping the image into blocks and coding each one. The difference is in how flexibly they do it.

JPEG uses fixed 8x8 blocks and nothing else. A cloudless sky and a face full of freckles get carved into the same rigid grid, and every block pays the same structural overhead.

HEVC divides the image into Coding Tree Units of 64x64 pixels, then recursively splits each one in a quadtree, where any square can split into four smaller squares all the way down to 4x4. AV1 goes further, with superblocks up to 128x128 and ten partition shapes including rectangles, again down to 4x4.

Why this matters is simple economics. A flat sky becomes a handful of huge, nearly free blocks; the coding budget flows to the eyelashes and the brickwork instead. JPEG has no way to make that trade, and spends the same overhead on emptiness as on detail.

Intra prediction, the big idea JPEG lacks

This is the heart of the whole approach. Before compressing a block, the codec predicts it from the already-decoded pixels along its top and left edges, then codes only the residual, meaning the difference between the prediction and reality. If the prediction is good, the residual is close to zero and costs almost nothing to store.

Think of it as the decoder being told that a block continues the gradient above it, plus a small correction, instead of being handed the block from scratch. JPEG has no equivalent and codes every block from scratch.

HEVC offers 35 prediction modes: planar (smooth gradients), DC (flat fill), and 33 directional modes that extend edges through the block at various angles. AV1 expands the directional set to 56 variants and adds a few specialty modes.

• Chroma-from-luma prediction derives the color planes from the already-decoded brightness plane. In most photos color edges follow brightness edges, so it works well.
• Palette mode stores a short color list and per-pixel indices instead of transformed residuals, which suits graphics and text with few distinct colors.
• Intra block copy reuses a repeated region from elsewhere in the same image, so a UI element that appears many times in a screenshot is coded once and referenced thereafter.

Bigger transforms make smoother gradients

All three formats then transform their blocks into frequency space and quantize the coefficients, a part of the recipe that hasn't changed since 1992. What changed is the toolbox. JPEG has exactly one transform, the 8x8 DCT. HEVC uses the DCT (plus a DST for small blocks) at sizes from 4x4 up to 32x32. AV1 offers DCT, ADST, flipped ADST, and identity transforms, in rectangular shapes, at sizes up to 64x64.

JPEG's signature failures live at this stage. A smooth gradient across a large area, described in 8x8 pieces, degenerates into a staircase of per-block averages, which is the banding you see in skies, and the visible seams between the pieces are the blocking. A 64x64 transform describes the same gradient in a few coefficients, in one piece.

Entropy coding recovers the last few percent

After quantization, everything gets squeezed into the final bitstream. JPEG uses Huffman coding, which relies on static tables and charges a whole number of bits for every symbol. If a symbol's ideal cost is 0.3 bits, Huffman charges a full bit anyway, a rounding tax collected on millions of symbols per image.

HEVC uses CABAC, short for context-adaptive binary arithmetic coding. Arithmetic coding narrows a single number's range with each symbol rather than emitting whole bits, so a very likely symbol can genuinely cost a fraction of a bit. The "context-adaptive" part means the probability estimates update continuously as the image is coded, so the coder learns your image while compressing it instead of consulting a fixed table designed decades ago. AV1 uses a multi-symbol arithmetic coder built on the same idea. The payoff is roughly 5 to 15% smaller output with no loss at all, purely from smarter bookkeeping.

In-loop filters keep artifacts from compounding

JPEG has nothing at this stage. Whatever block seams and ringing the quantizer creates are simply in the picture.

The video codecs filter inside the coding loop. HEVC applies a deblocking filter plus SAO (sample adaptive offset); AV1 applies deblocking plus CDEF (a constrained directional enhancement filter that cleans ringing along edges) plus loop restoration. Because these run inside the codec, intra prediction works from the cleaned-up pixels — each block predicts from filtered neighbors rather than from artifacts, so errors don't compound across the image. It's the difference between fixing mistakes as you go and photographing the mess afterward.

Film grain synthesis

AV1 has one audacious tool with no HEVC or JPEG equivalent. Noise, like the sensor grain in a high-ISO photo, is nearly incompressible because it is random, so every codec either spends a fortune preserving it or smears it away. AV1 refuses both options. It removes the grain, encodes the clean image cheaply, describes the grain statistically in a few bytes, and re-synthesizes fresh grain of matching character on decode. The pixels of noise you see aren't the pixels the camera captured, just statistically indistinguishable replacements. On high-ISO photos this saves a lot of space.

Bit depth and color

JPEG is, in practice, 8 bits per channel with 4:2:0 chroma subsampling. HEIC supports 8- and 10-bit. AVIF supports 8-, 10-, and 12-bit, chroma at 4:2:0, 4:2:2, or full 4:4:4, plus HDR transfer functions (PQ, HLG) and wide color gamuts signaled through CICP.

Ten bits matter even if you never touch HDR. Every edit, whether a curves adjustment or a shadow lift, stretches some part of the tonal range, and 8-bit gradients tear into visible bands when stretched. Two extra bits mean four times the tonal resolution — headroom that keeps skies smooth through editing as well as at capture.

The pipeline at a glance

The licensing economics

Technically, AV1 builds on HEVC's ideas with more modes, bigger blocks, and cleverer filters. The larger change is in the licensing. HEVC is covered by thousands of patents administered by multiple competing pools with terms that browser vendors refused to accept; that, more than any technical shortcoming, is why HEIC never reached the web. AV1 was built under the Alliance for Open Media's patent cross-license, royalty-free by charter, with members including Google, Apple, Netflix, Mozilla, Amazon, Microsoft, Intel, and Cisco. Alongside a better codec, the industry built a legal structure in which the codec could actually ship everywhere.

What all this cleverness costs

None of it is free. Searching dozens of prediction modes across ten partition shapes makes encoding far more expensive than JPEG's single pass. Phones mitigate this with dedicated hardware encoders, and software encoders offer speed presets that trade compression for time. Decoding is heavier too, which ancient devices feel. There is no progressive rendering, so you never get the blurry-to-sharp preview that progressive JPEG shows while a file loads. And very large images are encoded as tiled grids of independent sub-images, which early encoders occasionally betrayed with faint seam artifacts along tile boundaries.

These are reasonable trades, but it is worth knowing that the halved file sizes were paid for with complexity.