gdal raster tile reduced quality when compared to gdal2tiles (Lanczos in presence of nodata)

[PSZ-Code]

What is the bug?

gdal2tiles tested on GDAL 3.10
gdal raster tile tested on GDAL 3.12

I used to run gdal2tiles to generate a clear view of georeferenced vector pdf's. When testing gdal raster tile, i notice a decrease in quality (but also a significant increase in speed).
Quality for this case is defined as sharpness & resemblance to the original pdf, as well as the appearance of artifacts (in this case mostly white pixels between the initial image and transparant zones)

Input is a georeferenced pdf, containing a grey color which is recognized as the nodata value and georeferenced in a local coordinate system (thus the code must reproject)

I've tested 5 methods. For each case using "gdal raster tile", i find the quality to be significantly lower than gdal2tiles.
Note that gdal raster tile uses 'average' resampling, where 'gdal2tiles' used 'antialias'. This is because 'lanczos' in gdal raster tile returned awfull results

Test 1 (Base test):
gdal2tiles -r antialias -z 16-20 -a 178 input.pdf outputfolder --config GDAL_PDF_DPI 600
--> Clear and smooth image, almost no white artifacts at text edges

Test 2 lanczos:
gdal raster tile --min-zoom 16 --max-zoom 20 -r lanczos --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
--> Awful, a lot of lost data. I had expected similar results as the base test

Test 3 average:
gdal raster tile --min-zoom 16 --max-zoom 20 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
--> to my suprise, average resampling returned the best results of all resampling methods, yet far below the base test. Text became a lot 'fatter'
Setting a higher DPÏ does not improve results

Test 4: generating additional zoom level and averaging from there
gdal raster tile --min-zoom 16 --max-zoom 21 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
--> Better, but still artifacts and decreased sharpness compared to base test

Test 5: increase tile width to 512
gdal raster tile --min-zoom 16 --max-zoom 20 --tile-size 512 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
--> Closest to base test, especially for text, but significant increase in tile size and more 'corny' than original antialias

It seems the new implementation has significantly decreased quality, possibly in the reprojection. Especially the Lanczos (which i understood was used for antialias) algorithm returns significant different results

Running gdal2tiles on 3.12 with antialias returns similar results as gdal raster tile. I assume somewhere in these versions the use of PIL was removed...?

Steps to reproduce the issue

Comparison of following codes on geopdf in local coordinate system

GDAL3.10
gdal2tiles -r antialias -z 16-20 -a 178 input.pdf outputfolder --config GDAL_PDF_DPI 600

GDAL 3.12
gdal raster tile --min-zoom 16 --max-zoom 20 -r lanczos --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
gdal raster tile --min-zoom 16 --max-zoom 20 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
gdal raster tile --min-zoom 16 --max-zoom 21 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600
gdal raster tile --min-zoom 16 --max-zoom 20 --tile-size 512 -r average --skip-blank input.pdf outputfolder --config GDAL_PDF_DPI 600

Versions and provenance

Windows 11
gdal 3.10 for gdal2tiles
gdal 3.12 for gdal raster tile

Additional context

No response

[jratike80]

Where is the PDF to test with?

[PSZ-Code]

The pdf can not be shared due to proprietary content.

[rouault]

@PSZ-Code We need something to be able to reproduce (like a clipped version of the PDF that can be generate with gdal raster clip)

[rouault]

Running gdal2tiles on 3.12 with antialias returns similar results as gdal raster tile. I assume somewhere in these versions the use of PIL was removed...?

yes, in 22a22f8

[PSZ-Code]

Sample attached
sample.pdf
sample.pdf.aux.xml

[jratike80]

With --bilinear the result looks rather good.

[PSZ-Code]

The result will indeed look quite alright, but experience from similar maps has shown me that lanczos often returns the most satisfiable result, especially when generating the overview tiles.
In addition to the base tiles being of lower quality, using lanczos for just the overview tiles also causes a lot of color distortion (e.g. text becoming black), which also didn't occur in the old implementation.

The old implementation of lanczos (using PIL) was, for me, far superior in quality for the base tiles, which of course impacts the overview tiles (which I generally generate myself with PIL)

@rouault Still very grateful for the significant speed improvements and library as-is

[jratike80]

(e.g. text becoming black)

I wonder if there could there be a difference in how PIL and GDAL resamples the nodata that exists all around the texts. By deleting the pam.xml file the result from gdal tile is nice and sharp, but transparency is naturally lost.

[rouault]

in #14572 a set of fixes that largely improve the situation, without fully fixing the problem. The real difficulty would be to find how to code Lanczos resampling in presence of massive amount of nodata pixels around the text, which is the case here. The source image has antialiased rendering of text on a gray background, so some non transparent pixels capture greyish tones. The "ideal" fix would perhaps have to "revert" that and transport the source image to the target transparent color before doing the resampling... I don't know...