Seeking help with custom data

[Wang-Wenqing]

When I use custom data to generate videos, I sometimes get poor results. Could you please provide some guidance on how to improve them?

if i give a input image like this:

first step: VGGT — 3D scene warping (single / few images)
with command:

python run_warp.py \
    --image_path image_1 \
    --output_path output_images_1_vggt_warp_degree90 \
    --camera 0 \
    --direction right \
    --degree 90 \
    --frame_single 25 \
    --look_at_depth 0.25

and then Prepare Prompts
"city_square": (
"A bullet-time effect video in a frozen 3D photography style. The entire small-town courthouse-square intersection is captured as a single, perfectly static moment in time. The red-brick civic building with a clock tower, the surrounding storefronts and awnings, the gazebo, the roads, cars, pedestrians, flag, trees, shadows, and warm afternoon atmosphere all remain completely motionless, with no movement anywhere in the environment. The only change is the camera, which moves smoothly and stably in a gentle aerial arc while maintaining a high-angle perspective. The scene should remain a coherent small-town urban center throughout the shot: newly revealed areas should continue the courthouse-square layout with more connected streets, sidewalks, rooftops, storefronts, parked vehicles, and town-block structures consistent with the existing intersection. The surrounding view should preserve the continuity of the town rather than collapsing into generic woodland or empty natural scenery."
),

and then Video Inference with command:

#!/bin/bash
# WorldForge (WAN) Video Generation - Batch Inference Script
# Usage: bash wan_for_worldforge/run_test_case.sh  (run from project root)
# export HF_ENDPOINT=https://hf-mirror.com  # Uncomment if you need a HuggingFace mirror
cd "$(dirname "$0")"
# ==================== Basic Configuration ====================
MODELS_DIR="/.cache/huggingface/models"        # Model weights directory (must contain Wan2.1-I2V-14B-480P-Diffusers etc.)
VIDEO_REF="vggt/output_images_1_vggt_warp_degree90/warped_images"            # Input frames + masks directory
OUTPUT_DIR="./output_image_1_wan_worldforge_degree90"                          # Output directory
SCENE="city_square"                                   # Scene name (see utils/prompts.py)
NUM_FRAMES=49                                  # Number of output frames
RESOLUTION="720p"                              # 480p or 720p
STATIC="True"                                 # True for static scenes, False for dynamic
NUM_INFERENCE_STEPS=50                         # Diffusion sampling steps

# ==================== Parameter Grid ====================
# Modify these arrays to sweep different parameter combinations
omegas=(4)                  # Auto-guidance strength (recommended: (4 6)) 
guidance_scales=(4)         # CFG scale (recommended: (4) ) 
transition_distances=(15)   # Mask softening distance in pixels (0=hard edge; recommended: (15 20 25))
resample_steps=(2)          # Resampling iterations per step (recommended: (2))
guide_steps=(10 18 23)         # Guide steps: apply guided fusion for first N steps (recommended: (10 15 18 20 23))
step_additions=(0)        # Extra steps for resample_round = guide_steps + addition (recommended: (0 1))

# ==================== Batch Inference ====================
mkdir -p "$OUTPUT_DIR"

for omega in "${omegas[@]}"; do
for cfg in "${guidance_scales[@]}"; do
for mask in "${transition_distances[@]}"; do
for resample in "${resample_steps[@]}"; do
for guide in "${guide_steps[@]}"; do
for add in "${step_additions[@]}"; do
    round=$((guide + add))
    output="${OUTPUT_DIR}/o${omega}_guide${guide}_round${round}_mask${mask}_cfg${cfg}.mp4"
    
    echo "========================================"
    echo "omega=$omega, guide=$guide, round=$round, mask=$mask, cfg=$cfg"
    echo "output: $output"
    echo "========================================"
    
    python infer_worldforge.py \
        --model "$RESOLUTION" \
        --models-dir "$MODELS_DIR" \
        --video-ref "$VIDEO_REF" \
        --scene "$SCENE" \
        --num-frames $NUM_FRAMES \
        --num-inference-steps $NUM_INFERENCE_STEPS \
        --guidance-scale $cfg \
        --static "$STATIC" \
        --guided \
        --resample-steps $resample \
        --guide-steps $guide \
        --resample-round $round \
        --omega $omega \
        --omega_resample $omega \
        --soften-mask \
        --transition-distance $mask \
        --use-pca-channel-selection \
        --output "$output"
done
done
done
done
done
done

but get results very weird no matter which guide_steps

o4_guide23_round23_mask15_cfg4.mp4

[Chenxi-Song]

Hi Wenqing,

Thanks for trying out WorldForge!

The issue you are experiencing is mainly because the warping angle in a single pass is too large.

If you want to generate a large-angle scene, I highly recommend breaking it down into smaller, iterative steps:

Generate about 20-25 degrees per inference pass.

Set a slightly higher frame count (e.g., around 37 or 41 frames).

Take the last frame of this generated video and use it as the starting input image for the next inference pass to continue the camera movement.

While this iterative approach takes a bit more time, it should guarantee much better generation quality.

(PS: We are aware of the long inference times and the hassle of parameter tuning. These issues will be addressed in our upcoming V2 release. Please stay tuned if you are interested in our work!)

Let me know if you have any other questions.

[Wang-Wenqing]

Thank you for your advice. I followed your suggestions and ran the code as recommended, but the results are still a bit inconsistent. I was wondering whether you might have any further advice or suggestions.

set -euo pipefail

# ==================== Path Configuration ====================
PROJECT_ROOT="$(cd "$(dirname "$0")" && pwd)"
MODELS_DIR=".cache/huggingface/models"

# ==================== Iterative Pipeline Configuration ====================
INITIAL_IMAGE_DIR="image_1"
DIRECTION="right"          # Camera movement direction: left/right/up/down/forward/backward
DEGREE_PER_PASS=22.5       # Degrees warped per pass (20-25 recommended for smooth continuity)
NUM_FRAMES=41              # Frames per generated video (37 or 41 recommended)
LOOK_AT_DEPTH=0.25         # Focus depth coefficient (same as original)
NUM_PASSES=4               # Total number of iterative passes

OUTPUT_BASE_DIR="${PROJECT_ROOT}/output_iterative_${DIRECTION}_${DEGREE_PER_PASS}deg"

# ==================== WAN Inference Parameters ====================
SCENE="city_square"
RESOLUTION="720p"
STATIC="True"
NUM_INFERENCE_STEPS=50
OMEGA=4
GUIDANCE_SCALE=4
TRANSITION_DISTANCE=15
RESAMPLE_STEPS=2
GUIDE_STEPS=18
RESAMPLE_ROUND=18          # Typically GUIDE_STEPS + 0 (or +1 for extra refinement)

# ==================== Helpers ====================
log() { echo "[$(date '+%H:%M:%S')] $*"; }
separator() { echo ""; echo "========================================"; }

# ==================== Main Loop ====================
mkdir -p "$OUTPUT_BASE_DIR"

# Write run config for reproducibility
cat > "${OUTPUT_BASE_DIR}/run_config.txt" << EOF
Run started: $(date)
INITIAL_IMAGE_DIR=$INITIAL_IMAGE_DIR
DIRECTION=$DIRECTION
DEGREE_PER_PASS=$DEGREE_PER_PASS
NUM_FRAMES=$NUM_FRAMES
NUM_PASSES=$NUM_PASSES
LOOK_AT_DEPTH=$LOOK_AT_DEPTH
SCENE=$SCENE
RESOLUTION=$RESOLUTION
OMEGA=$OMEGA
GUIDANCE_SCALE=$GUIDANCE_SCALE
TRANSITION_DISTANCE=$TRANSITION_DISTANCE
RESAMPLE_STEPS=$RESAMPLE_STEPS
GUIDE_STEPS=$GUIDE_STEPS
RESAMPLE_ROUND=$RESAMPLE_ROUND
NUM_INFERENCE_STEPS=$NUM_INFERENCE_STEPS
EOF

CURRENT_IMAGE_DIR="$INITIAL_IMAGE_DIR"
TOTAL_DEGREES=0

for pass in $(seq 1 "$NUM_PASSES"); do
    separator
    TOTAL_DEGREES=$(awk "BEGIN {printf \"%.1f\", $TOTAL_DEGREES + $DEGREE_PER_PASS}")
    log "PASS $pass / $NUM_PASSES  |  camera total: ~${TOTAL_DEGREES} deg ${DIRECTION}"
    log "Input image dir: $CURRENT_IMAGE_DIR"
    separator

    PASS_DIR="${OUTPUT_BASE_DIR}/pass${pass}"
    mkdir -p "$PASS_DIR"

    # ── Step 1: VGGT Warping ─────────────────────────────────────────────────
    WARP_OUTPUT="${PASS_DIR}/vggt_warp"
    log "[Pass $pass | Step 1/3] VGGT warping  (direction=${DIRECTION}, degree=${DEGREE_PER_PASS}, frames=${NUM_FRAMES})"

    cd "${PROJECT_ROOT}/vggt"
    python run_warp.py \
        --image_path "$CURRENT_IMAGE_DIR" \
        --output_path "$WARP_OUTPUT" \
        --camera 0 \
        --direction "$DIRECTION" \
        --degree "$DEGREE_PER_PASS" \
        --frame_single "$NUM_FRAMES" \
        --look_at_depth "$LOOK_AT_DEPTH"

    log "[Pass $pass | Step 1/3] Done → ${WARP_OUTPUT}/warped_images"

    # ── Step 2: WAN WorldForge Inference ─────────────────────────────────────
    VIDEO_REF="${WARP_OUTPUT}/warped_images"
    VIDEO_OUTPUT="${PASS_DIR}/o${OMEGA}_guide${GUIDE_STEPS}_round${RESAMPLE_ROUND}_mask${TRANSITION_DISTANCE}_cfg${GUIDANCE_SCALE}.mp4"

    log "[Pass $pass | Step 2/3] WAN WorldForge inference  (resolution=${RESOLUTION}, steps=${NUM_INFERENCE_STEPS})"

    cd "${PROJECT_ROOT}/wan_for_worldforge"
    python infer_worldforge.py \
        --model "$RESOLUTION" \
        --models-dir "$MODELS_DIR" \
        --video-ref "$VIDEO_REF" \
        --scene "$SCENE" \
        --num-frames "$NUM_FRAMES" \
        --num-inference-steps "$NUM_INFERENCE_STEPS" \
        --guidance-scale "$GUIDANCE_SCALE" \
        --static "$STATIC" \
        --guided \
        --resample-steps "$RESAMPLE_STEPS" \
        --guide-steps "$GUIDE_STEPS" \
        --resample-round "$RESAMPLE_ROUND" \
        --omega "$OMEGA" \
        --omega_resample "$OMEGA" \
        --soften-mask \
        --transition-distance "$TRANSITION_DISTANCE" \
        --use-pca-channel-selection \
        --output "$VIDEO_OUTPUT"

    log "[Pass $pass | Step 2/3] Done → $VIDEO_OUTPUT"

    # ── Step 3: Extract Last Frame ───────────────────────────────────────────
    NEXT_IMAGE_DIR="${PASS_DIR}/last_frame_image"
    mkdir -p "$NEXT_IMAGE_DIR"
    LAST_FRAME_PATH="${NEXT_IMAGE_DIR}/image.png"

    log "[Pass $pass | Step 3/3] Extracting last frame (frame index $((NUM_FRAMES - 1)))..."

    # Select frame by 0-based index — reliable since we know the exact frame count
    LAST_IDX=$((NUM_FRAMES - 1))
    ffmpeg -i "$VIDEO_OUTPUT" \
        -vf "select=eq(n\,${LAST_IDX})" \
        -vsync vfr \
        -frames:v 1 \
        "$LAST_FRAME_PATH" \
        -y 2>&1 | tail -3

    if [ ! -f "$LAST_FRAME_PATH" ]; then
        log "WARNING: select filter did not produce a frame; falling back to sseof extraction."
        ffmpeg -sseof -2 -i "$VIDEO_OUTPUT" -frames:v 1 -update 1 "$LAST_FRAME_PATH" -y
    fi

    log "[Pass $pass | Step 3/3] Last frame saved → $LAST_FRAME_PATH"

    # Record per-pass summary
    echo "pass=${pass}, degree_total=${TOTAL_DEGREES}, video=${VIDEO_OUTPUT}, last_frame=${LAST_FRAME_PATH}" \
        >> "${OUTPUT_BASE_DIR}/run_config.txt"

    # Advance to next pass
    CURRENT_IMAGE_DIR="$NEXT_IMAGE_DIR"

    separator
    log "Pass $pass complete."
done

separator
echo ""
log "ALL $NUM_PASSES PASSES COMPLETE"
log "Total camera movement: ~${TOTAL_DEGREES} degrees ${DIRECTION}"
log "Results in: $OUTPUT_BASE_DIR"
echo ""
log "Pass summary:"
for pass in $(seq 1 "$NUM_PASSES"); do
    PASS_DIR="${OUTPUT_BASE_DIR}/pass${pass}"
    VIDEO=$(ls "${PASS_DIR}"/*.mp4 2>/dev/null | head -1 || echo "(missing)")
    echo "  Pass $pass → $VIDEO"
done
separator

o4_guide18_round18_mask15_cfg4.mp4

o4_guide18_round18_mask15_cfg4.mp4

o4_guide18_round18_mask15_cfg4.mp4

o4_guide18_round18_mask15_cfg4.mp4

[Chenxi-Song]

Hi Wenqing,

Since our method operates entirely at inference time, the optimal parameters will vary significantly from scene to scene. Unfortunately, there is no "one-size-fits-all" configuration. You will need to test a few combinations and cherry-pick the best one for your specific custom image—this is exactly why we included the nested for loops in the run_test_case.sh script.

Looking at your script, you are currently using a fixed set of parameters:
OMEGA=4, GUIDANCE_SCALE=4, TRANSITION_DISTANCE=15, RESAMPLE_STEPS=2, GUIDE_STEPS=18, RESAMPLE_ROUND=18

I highly recommend expanding your search space to something like this:

OMEGA: 4 or 6

GUIDANCE_SCALE: 4

TRANSITION_DISTANCE: 15 or 20

RESAMPLE_STEPS: 2

GUIDE_STEPS: 10, 13, 15, or 18

RESAMPLE_ROUND: guide_steps + 0 or guide_steps + 1

By sweeping through these combinations, you have a much higher chance of finding the "sweet spot" that generates the best quality for your specific data.