Skip to content
This repository was archived by the owner on Nov 3, 2025. It is now read-only.
This repository was archived by the owner on Nov 3, 2025. It is now read-only.

to-tf-od: rework shard output #36

Description

@fracpete

Currently, the -s option allows one to specify the shard files to generate. However, this is not feasible when creating hundreds of shard files.
Instead have the following two options:

  • --num_shards <int>: how many shard files to generate, default is -1, so no sharding
  • --format <format>: the format for the file names (see below)
  • --width <int>: the number of digits to use to the index/total number of shards in the file name (left-padded with 0s), default is 5

The shard format string supports the following two placeholders:

  • {INDEX} - the 1-based index for the shard file
  • {TOTAL} - the total number of shard files that get generated (see --num_shards)

This allows the specification of formats:

  • --format "train.record-{INDEX}-{TOTAL}" (see here)
  • --format "train-{INDEX}-{TOTAL}.tfrecords"

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions