Merging pages causes corrupt output (Filter FlateDecode is set, but content is unencoded)

[larsga]

I'm stamping a template onto an existing document. The existing document was also produced by PyPDF, but by a different organization. The input PDF has issues, according to different tools I've used to look at it.

I'm reading the input pages one by one, then stamping onto each a page from a different PDF. The fourth page is displayed as blank, and the reason is that the PDF for that page is corrupt. If I skip the first three pages the fourth page is OK.

Environment

I'm using Python 3.12.3 on macOS-12.7.1-x86_64-i386-64bit. PyPDF is installed via pip:

pypdf==5.4.0, crypt_provider=('cryptography', '42.0.8'), PIL=10.3.0

Code + PDF

This reproduces the problem:

import pypdf

origin = pypdf.PdfReader('1.pdf')
template = pypdf.PdfReader('MinimalJob.niso.xml.stamp.pdf')
writer = pypdf.PdfWriter()

for ix in range(len(origin.pages)):
    page = origin.get_page(ix)
    stamp = template.get_page(1)
    page.merge_page(stamp, False, True)
    writer.add_page(page)

writer.write('out.pdf')

The two input files are attached. I'm checking for permission to add those to your tests. Will come back to you on that.

1.pdf

MinimalJob.niso.xml.stamp.pdf

Traceback

There is no traceback. PyPDF is perfectly happy to output the file, but if I try to read that file with MuPDF I get:

MuPDF error: library error: zlib error: incorrect header check

A PDF comparison tool (Antenna House Regression Test Suite) chokes on the output PDF.

MacOS Preview displays the last page as empty, because of the corruption issue.

pdftk refuses to process the file, because it finds errors.

[larsga]

This PDF is OK to include in your test suite. I checked, and it still reproduces the issue.

1.pdf

[stefan6419846]

Thanks for the report. Strangely enough, this does not seem to occur when using clone_from instead:

import pypdf

template = pypdf.PdfReader('MinimalJob.niso.xml.stamp.pdf')
writer = pypdf.PdfWriter(clone_from='1.pdf')

stamp = template.pages[1]
for page in writer.pages:
    page.merge_page(stamp, False, True)

writer.write('out.pdf')

Further digging into this shows that skipping the TOC page with your approach (ix == 2) leads to the correct output as well. This indicates that there is some side effect leading to the last page being considered empty.

The culprit most likely is that the generated content stream declares a /FlateDecode filter (which usually relies on zlib, matching your error message) without having actually applied it:

53 0 obj
<<
/Filter /FlateDecode
/Length 17637
>>
stream
q
BT
/Artifact <</Attached [/Top ]/BBox [442.8 794.0519 561 810.204 ]/Subtype /Header /Type /Pagination >>BDC 
/CS0 cs 0  scn
/TT0 1 Tf
...

The corresponding input stream looks correct:

58 0 obj
<<
/Filter /FlateDecode
/Length 4831
>>
stream
H�€W[oã...

Further analysis is required here on why the output generates this filter mismatch.

[larsga]

Yes, maybe I should have added that clone_from works. I had limited time, unfortunately.

That, however, triggers a different bug on some of our test cases. I haven't had time to report that one.

I managed to get the above to work by doing:

                current_page_orig = current_pdf.get_page(page_index)

                current_page = current_page_orig.create_blank_page(
                    target,
                    width = current_page_orig.mediabox.width,
                    height = current_page_orig.mediabox.height
                )
                current_page.merge_page(stamp, False, True)
                current_page.merge_page(current_page_orig)
                writer.add_page(current_page)

[stefan6419846]

I did some further debugging on this and how we end up with decoded data with a filter specified, but without spending more time, I could not identify the offending code section yet.