Skip to content

Commit 4e93bbc

Browse files
fdintinofdintino
committed
feat: gif->gif memory usage optimizations; add gifski option
1 parent 286be4f commit 4e93bbc

13 files changed

Lines changed: 1503 additions & 55 deletions

‎.github/workflows/test.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ jobs:
4141
sudo make install
4242
rm -rf gifsicle-${{ env.GIFSICLE_VER }}
4343
44+
- name: Install gifski
45+
run: |
46+
cargo install gifski
47+
4448
- name: Install system dependencies
4549
run: |
4650
sudo apt-get update

‎CHANGELOG.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,25 @@ Changelog
33

44
**Unreleased**
55

6+
* Fixed: animated-gif transcodes no longer exhaust memory on large inputs.
7+
``probe()`` now reads gif size and duration from a single-pass header parser
8+
(``thumbor_video_engine.utils.parse_gif``) instead of decoding every frame
9+
with Pillow, and ``transcode_to_gif()`` applies all geometry in ffmpeg at the
10+
target size with on-disk intermediates. When ``FFMPEG_USE_GIFSICLE_ENGINE`` is
11+
enabled, gifsicle now runs as a geometry-free ``-O3`` optimization pass
12+
file-to-file rather than loading the full-resolution animation into thumbor's
13+
gif engine. Python heap usage is now bounded regardless of source resolution
14+
or frame count.
15+
* Feature: optional streaming ``gifski`` pipeline, enabled with
16+
``FFMPEG_GIF_PIPELINE = 'gifski'`` (plus ``GIFSKI_PATH``, ``GIFSKI_QUALITY``,
17+
``GIFSKI_MAX_TARGET_PIXELS``, ``GIFSKI_GIFSICLE_PASS``). Streams frames from
18+
ffmpeg into gifski at the target size for faster, higher-quality gifs, falling
19+
back to the legacy pipeline for variable-delay gifs, oversized targets, or when
20+
the gifski binary is absent. gifski is AGPL-3.0 and is never a hard dependency;
21+
it is invoked as an unmodified subprocess only when configured.
22+
* Feature: ``MAX_ANIMATION_PIXELS`` rejects animations whose
23+
``width * height * frame_count`` exceeds the limit with a ``400`` response
24+
(disabled by default).
625
* Drop support for python 2.7 and thumbor 6
726

827
**1.2.5 (Jul 8, 2024)**

‎docs/configuration.rst‎

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,11 @@ FFMPEG\_USE\_GIFSICLE\_ENGINE
5858
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5959

6060
Equivalent to USE\_GIFSICLE\_ENGINE, but for the FFmpeg engine. It defaults to
61-
``False``. If ``True``, it will perform any image operations on animated gifs
62-
(e.g. cropping and resizing) using gifsicle (by way of ``GIF_ENGINE``).
61+
``False``. If ``True``, gifsicle runs as a final ``-O3`` optimization pass
62+
(plus ``GIFSICLE_ARGS``) over the gif produced by ffmpeg, reducing file size.
63+
All geometry (cropping, resizing) is applied by ffmpeg at the target size, so
64+
the gifsicle pass performs no resizing of its own and runs file-to-file on
65+
disk — the full animation is never buffered in the Python heap.
6366

6467
FFMPEG\_HANDLE\_ANIMATED\_GIF
6568
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -88,6 +91,80 @@ Specifies whether H265 format should be used automatically if the
8891
source image is an animated gif and the request accepts it (via
8992
``Accept: video/*``). It defaults to ``False``.
9093

94+
FFMPEG\_GIF\_PIPELINE
95+
~~~~~~~~~~~~~~~~~~~~~
96+
97+
Selects the gif-to-gif transcode pipeline. It defaults to ``'legacy'``.
98+
99+
``'legacy'``
100+
The ``palettegen``/``paletteuse`` pipeline. ffmpeg applies all geometry
101+
(crop, resize) at the **target** size, writes intermediates to disk, and
102+
— when ``FFMPEG_USE_GIFSICLE_ENGINE`` is enabled — runs a final
103+
geometry-free ``gifsicle -O3`` optimization pass. Only the final output
104+
bytes ever enter the Python heap, so memory stays bounded regardless of
105+
the source animation's resolution or frame count.
106+
107+
``'gifski'``
108+
Streams frames from ffmpeg directly into the `gifski`__ encoder at the
109+
target size, producing noticeably higher-quality gifs much faster. Inputs
110+
that gifski cannot represent are routed back to the ``legacy`` path
111+
automatically:
112+
113+
- **variable per-frame delays** (gifski emits a constant frame rate),
114+
- **target sizes above** ``GIFSKI_MAX_TARGET_PIXELS``,
115+
- and any request when the ``gifski`` binary is not available.
116+
117+
Visibly-transparent gifs are decoded to PNG frames first (gifski
118+
preserves alpha from PNG input); opaque gifs and video sources stream
119+
through a ``yuv4mpegpipe``.
120+
121+
.. note::
122+
gifski is licensed under the `AGPL-3.0`__. thumbor-video-engine
123+
invokes it as an unmodified subprocess (aggregation, not linking) and
124+
never declares it as a dependency. You must install the ``gifski``
125+
binary yourself to use this pipeline, and your deployment is
126+
responsible for AGPL compliance.
127+
128+
.. __: https://gif.ski/
129+
.. __: https://www.gnu.org/licenses/agpl-3.0.html
130+
131+
GIFSKI\_PATH
132+
~~~~~~~~~~~~
133+
134+
Path to the gifski binary. It defaults to ``None``, in which case gifski is
135+
looked up on ``PATH``. Only used when ``FFMPEG_GIF_PIPELINE`` is ``'gifski'``.
136+
137+
GIFSKI\_QUALITY
138+
~~~~~~~~~~~~~~~
139+
140+
Quality (1–100) passed to gifski (``--quality``). Defaults to ``90``.
141+
142+
GIFSKI\_MAX\_TARGET\_PIXELS
143+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
144+
145+
Above this output size (target ``width * height``), the gifski pipeline routes
146+
through the legacy path instead. gifski's quantizer working set grows with
147+
output dimensions, while the legacy path's memory stays bounded — this trades
148+
a bit of wall time for bounded subprocess memory on large outputs. Defaults to
149+
``1440000`` (1600×900). A value of ``0`` disables the switch.
150+
151+
GIFSKI\_GIFSICLE\_PASS
152+
~~~~~~~~~~~~~~~~~~~~~~
153+
154+
If ``True``, run a final geometry-free ``gifsicle -O3`` pass (plus
155+
``GIFSICLE_ARGS``, e.g. ``--lossy``) over gifski's output to further reduce
156+
file size. Defaults to ``False``.
157+
158+
MAX\_ANIMATION\_PIXELS
159+
~~~~~~~~~~~~~~~~~~~~~~
160+
161+
Maximum total pixels (``width * height * frame_count``) allowed for an animated
162+
gif. Animations over this limit fail with a ``400`` response. thumbor's
163+
``MAX_PIXELS`` is per-frame and does not bound frame count, so a high-frame-count
164+
gif can still consume large amounts of memory; this gate, evaluated cheaply at
165+
load time from a single-pass header parse (no frame decoding), bounds the total.
166+
Defaults to ``0``, which disables the check.
167+
91168
FFPROBE\_PATH
92169
~~~~~~~~~~~~~
93170

‎src/thumbor_video_engine/__init__.py‎

Lines changed: 53 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,59 @@
3434
'Equivalent to USE_GIFSICLE_ENGINE, but for the ffmpeg engine',
3535
'Video')
3636

37+
Config.define(
38+
'FFMPEG_GIF_PIPELINE',
39+
'legacy',
40+
"Which GIF->GIF transcode pipeline to use. 'legacy' uses the "
41+
"palettegen/paletteuse pipeline (scaling to the target size in ffmpeg, "
42+
"with on-disk intermediates and, when FFMPEG_USE_GIFSICLE_ENGINE is on, "
43+
"a final geometry-free gifsicle -O3 optimization pass). 'gifski' streams "
44+
"frames from ffmpeg into the gifski encoder at the target size, falling "
45+
"back to the legacy path for inputs gifski cannot represent (variable "
46+
"frame delays) or when the gifski binary is not available.",
47+
'Video')
48+
49+
Config.define(
50+
'GIFSKI_PATH',
51+
None,
52+
'Path to the gifski binary. If None, gifski is looked up on PATH. Only '
53+
'used when FFMPEG_GIF_PIPELINE is "gifski". Note: gifski is licensed '
54+
'under the AGPL-3.0; it is invoked as an unmodified subprocess and is '
55+
'never a hard dependency of this package.',
56+
'Video')
57+
58+
Config.define(
59+
'GIFSKI_QUALITY',
60+
90,
61+
'Quality (1-100) passed to gifski (--quality) when encoding GIFs',
62+
'Video')
63+
64+
Config.define(
65+
'GIFSKI_MAX_TARGET_PIXELS',
66+
1440000,
67+
'Above this output size (target width * height), the gifski pipeline '
68+
'routes through the legacy palettegen/paletteuse + gifsicle path '
69+
"instead. gifski's quantizer working set grows with output dimensions "
70+
"while the legacy path's memory stays bounded, at the cost of wall "
71+
'time. A value of 0 disables the switch. Defaults to 1440000 (1600x900).',
72+
'Video')
73+
74+
Config.define(
75+
'GIFSKI_GIFSICLE_PASS',
76+
False,
77+
'If True, run a final geometry-free gifsicle -O3 pass (plus '
78+
'GIFSICLE_ARGS, e.g. --lossy) over gifski output to reduce file size',
79+
'Video')
80+
81+
Config.define(
82+
'MAX_ANIMATION_PIXELS',
83+
0,
84+
'Maximum total pixels (width * height * frame_count) allowed for an '
85+
'animated GIF. Animations over this limit fail with a 400 response '
86+
"(thumbor's MAX_PIXELS is per-frame and does not bound frame count). A "
87+
'value of 0 (the default) disables the check.',
88+
'Video')
89+
3790
Config.define(
3891
'FFMPEG_HANDLE_ANIMATED_GIF',
3992
True,
@@ -60,12 +113,6 @@
60113
'Whether to use two-pass encoding for h264 in ffmpeg',
61114
'Video')
62115

63-
Config.define(
64-
'FFMPEG_H264_TWO_PASS',
65-
False,
66-
'Whether to use two-pass encoding for h264 in ffmpeg',
67-
'Video')
68-
69116
Config.define(
70117
'FFMPEG_H264_PRESET',
71118
None,

0 commit comments

Comments
 (0)