Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 25 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ TorchCodec is a Python library for decoding video and audio data into PyTorch
tensors, on CPU and CUDA GPU. It also supports video and audio encoding on CPU!
It aims to be fast, easy to use, and well integrated
into the PyTorch ecosystem. If you want to use PyTorch to train ML models on
videos and audio, TorchCodec is how you turn these into data.
videos and audio, or run inference, TorchCodec is how you turn these into data.

We achieve these capabilities through:

Expand All @@ -19,13 +19,13 @@ We achieve these capabilities through:
* Returning data as PyTorch tensors, ready to be fed into PyTorch transforms
or used directly to train models.

## Using TorchCodec
## Usage Examples

Here's a condensed summary of what you can do with TorchCodec. For more detailed
examples, [check out our
Below are some examples of what you can do with TorchCodec. For more detailed
examples and more use-cases, [check out our
documentation](https://meta-pytorch.org/torchcodec/stable/generated_examples/)!

#### Decoding
#### Video Decoding

```python
from torchcodec.decoders import VideoDecoder
Expand Down Expand Up @@ -61,41 +61,31 @@ decoder.get_frames_played_at(seconds=[0.5, 10.4])
# duration_seconds: tensor([0.0334, 0.0334], dtype=torch.float64)
```

#### Clip sampling
You can use the following snippet to generate a video with FFmpeg and try out
the `VideoDecoder`:

```python

from torchcodec.samplers import clips_at_regular_timestamps

clips_at_regular_timestamps(
decoder,
seconds_between_clip_starts=1.5,
num_frames_per_clip=4,
seconds_between_frames=0.1
)
# FrameBatch:
# data (shape): torch.Size([9, 4, 3, 270, 480])
# pts_seconds: tensor([[ 0.0000, 0.0667, 0.1668, 0.2669],
# [ 1.4681, 1.5682, 1.6683, 1.7684],
# [ 2.9696, 3.0697, 3.1698, 3.2699],
# ... (truncated), dtype=torch.float64)
# duration_seconds: tensor([[0.0334, 0.0334, 0.0334, 0.0334],
# [0.0334, 0.0334, 0.0334, 0.0334],
# [0.0334, 0.0334, 0.0334, 0.0334],
# ... (truncated), dtype=torch.float64)
```bash
ffmpeg -f lavfi -i testsrc2=size=640x400:duration=10:rate=25 /tmp/output_video.mp4
```

You can use the following snippet to generate a video with FFmpeg and tryout
TorchCodec:
#### Encoding

```bash
fontfile=/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono-Bold.ttf
output_video_file=/tmp/output_video.mp4
```python
from torchcodec.encoders import Encoder

ffmpeg -f lavfi -i \
color=size=640x400:duration=10:rate=25:color=blue \
-vf "drawtext=fontfile=${fontfile}:fontsize=30:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:text='Frame %{frame_num}'" \
${output_video_file}
encoder = Encoder()
video_stream = encoder.add_video(
height=height, width=width, frame_rate=frame_rate,
)
audio_stream = encoder.add_audio(
sample_rate=sample_rate, num_channels=num_channels,
)
with encoder.open_file("output.mp4"):
video_stream.add_frames(frames_batch_0)
audio_stream.add_samples(samples_batch_0)
video_stream.add_frames(frames_batch_1)
audio_stream.add_samples(samples_batch_1)
# ...
```

## Installing TorchCodec
Expand Down Expand Up @@ -220,22 +210,6 @@ format you want. Refer to Nvidia's GPU support matrix for more details
conda install -c conda-forge "torchcodec=*=*cuda*"
```

## Benchmark Results

The following was generated by running [our benchmark script](./benchmarks/decoders/generate_readme_data.py) on a lightly loaded 22-core machine with an Nvidia A100 with
5 [NVDEC decoders](https://docs.nvidia.com/video-technologies/video-codec-sdk/12.1/nvdec-application-note/index.html#).

![benchmark_results](./benchmarks/decoders/benchmark_readme_chart.png)

The top row is a [Mandelbrot](https://ffmpeg.org/ffmpeg-filters.html#mandelbrot) video
generated from FFmpeg that has a resolution of 1280x720 at 60 fps and is 120 seconds long.
The bottom row is [promotional video from NASA](https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4)
that has a resolution of 960x540 at 29.7 fps and is 206 seconds long. Both videos were
encoded with libx264 and yuv420p pixel format. All decoders, except for TorchVision, used FFmpeg 6.1.2. TorchVision used FFmpeg 4.2.2.

For TorchCodec, the "approx" label means that it was using [approximate mode](https://meta-pytorch.org/torchcodec/stable/generated_examples/decoding/approximate_mode.html)
for seeking.

## Contributing

We welcome contributions to TorchCodec! Please see our [contributing
Expand Down
Loading