Skip to content

Fix best_dev_checkpoint path #167

@wasertech

Description

@wasertech

Produced checkpoints are using absolute path where they should really be using relative ones.

Checkpoints FR STT 0.9

# ./best_dev_checkpoint
model_checkpoint_path: "/mnt/checkpoints/best_dev-221133"
all_model_checkpoint_paths: "/mnt/checkpoints/best_dev-221133"

Here is English checkpoint for reference.

# ./best_dev_checkpoint
model_checkpoint_path: "best_dev-3663881"
all_model_checkpoint_paths: "best_dev-3663881"

Fix should probably go into package.sh I think.

Related : coqui-ai/STT#2338
^ STT's lm_optimizer throws a NotFoundError from TF, since the path points to nowhere in transfer-learning configuration, as checkpoint path is /transfer-checkpoint/ instead of /mnt/checkpoints/.

To use an existing checkpoint, just ensure the `docker run` includes a mount such as:
`type=bind,src=PATH/TO/CHECKPOINTS,dst=/transfer-checkpoint`. Upon running, data
will be copied from that place.

if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.pb" ]; then
echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
cp -a /transfer-checkpoint/* /mnt/checkpoints/
fi;

That was for DS but with STT, we have access to --load_checkpoint_dir and --save_checkpoint_dir flags instead.

	if [ -f "/transfer-checkpoint/checkpoint" -a ! -f "/mnt/models/output_graph.tflite" ]; then
		echo "Using checkpoint from ${TRANSFER_CHECKPOINT}"
		# use --load_checkpoint_dir for transfer learning
		LOAD_CHECKPOINT_FROM="--load_checkpoint_dir /transfer-checkpoint --save_checkpoint_dir /mnt/checkpoints"
	else
		LOAD_CHECKPOINT_FROM="--checkpoint_dir /mnt/checkpoints/"
	fi;

Example from my branch stt140-cv9

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions