OutOfMemoryException is thrown when generating PDF with many pages each of which contains images, as all of the document model data is in-memory.

The issue is fixed with the new PdfStreamWriter API. As an example, you may see the attached demo to the following feedback item:
https://feedback.telerik.com/Project/184/Feedback/Details/213503-pdfprocessing-optimize-pdfstreamwriter-cache-in-order-to-reduce-memory-when-writ

Available in LIB version: 2017.1.320