Matte color is used for preblending images with some background color, using the SMask. Matte color is specified using the optional 'Matte' entry for the SMask object.

See PDF 1.7 specification, page 554-555:

Matte
array
(Optional; PDF 1.4) An array of component values specifying the matte color with which the image data in the parent image has been preblended. The array consists of nnumbers, where n is the number of components in the color space specified by the ColorSpace entry in the parent image’s image dictionary; the numbers must be valid color components in that color space. If this entry is absent, the image data is not preblended.