predup youtube playlists and tabs, and more...#374

Draft

galgeek wants to merge 85 commits intointernetarchive:masterfrom

galgeek:predup_type_playlist

Commits on Jul 14, 2025

initial commit
Barbara Miller
committed
formatting fix
Barbara Miller
committed
ruff format updates
Barbara Miller
committed
CONCAT
Barbara Miller
committed
variable VIDEO_DATA
Barbara Miller
committed
skip ternary op for now
Barbara Miller
committed
VIDEO_DATA_SOURCE
Barbara Miller
committed
use job_conf.get()
Barbara Miller
committed
make psycopg dependency optional
Barbara Miller
committed
containing_page_url_pattern update
Barbara Miller
committed
type hint get_video_captures
Barbara Miller
committed
ruff import block fix
Barbara Miller
committed
initial interface update
Barbara Miller
committed
VideoDataWrapper refined
Barbara Miller
committed
VideoDataClient, generalized
Barbara Miller
committed
keep it simple for now
Barbara Miller
committed
fix github ruff issues
Barbara Miller
committed
self._video_data in worker
Barbara Miller
committed
dataclass VideoDataRecord
Barbara Miller
committed
dataclass VideoCaptureRecord instead
Barbara Miller
committed
def create_video_capture_record minimally
Barbara Miller
committed
more new fields
Barbara Miller
committed
worker._video_data and seed_id mostly
Barbara Miller
committed
save video_record here?
Barbara Miller
committed
save_video_capture_record
Barbara Miller
committed

Commits on Jul 15, 2025

add get_recent_video_capture, mostly
Barbara Miller
committed
updates for QA deploy
Barbara Miller
committed

Commits on Jul 16, 2025

log urls we skipped adding to outlinks
Barbara Miller
committed
updates from qa deploy
Barbara Miller
committed
more updates post-QA-deploy
Barbara Miller
committed

Commits on Jul 18, 2025

even more updates post-QA-deploy
Barbara Miller
committed
add account_id to model's populate defaults
Barbara Miller
committed

Commits on Jul 21, 2025

more populate_defaults maybe
Barbara Miller
committed
use only last proxyrack endpoint for non-youtube-watch urls
Barbara Miller
committed

Commits on Jul 22, 2025

preserve pre-2025.07.21 mtime handling
Barbara Miller
committed

Commits on Jul 23, 2025

account_id first (alphabetical order?)
Barbara Miller
committed

Commits on Jul 25, 2025

skip creating video records in brozzler for now
Barbara Miller
committed
ruff and isort
Barbara Miller
committed
do use uv, not pip
Barbara Miller
committed
skip 3.14 for now with psycopg; try 3.13
Barbara Miller
committed
job schema tweak -- more info: https://github.com/pyeve/cerberus/issues/220
Barbara Miller
committed
one more tweak?
Barbara Miller
committed
un-update job_schema.yaml
galgeek
authored
make tests pass, even with account_id
Barbara Miller
committed
fix typo
Barbara Miller
committed
add account_id to frontier tests
Barbara Miller
committed
add account_id to job_schema (again)
Barbara Miller
committed
Merge branch 'master' into predup_type_playlist
galgeek
authored

Commits on Jul 26, 2025

partition_id, not account_id
Barbara Miller
committed
... except account_id for pg_query
Barbara Miller
committed
fix repeated key
Barbara Miller
committed

Commits on Jul 27, 2025

simplify job conf
Barbara Miller
committed

Commits on Jul 28, 2025

check captured_youtube_watch_pages not None
Barbara Miller
committed
skip psycopg.rows.scalar_row
Barbara Miller
committed
fix logged errors
Barbara Miller
committed
import psycopg only in class VideoDataClient
Barbara Miller
committed
import psycopg only at top of file
Barbara Miller
committed

Commits on Jul 29, 2025

query_tuple, check result for None
Barbara Miller
committed
tidying
Barbara Miller
committed

Commits on Jul 30, 2025

tidy psycopg imports, mostly
Barbara Miller
committed

Commits on Jul 31, 2025

predup query for should_ytdlp
Barbara Miller
committed
ruff'd worker.py
Barbara Miller
committed
fix formatting errors
Barbara Miller
committed

Commits on Aug 1, 2025

updates for get_video_capture mostly
Barbara Miller
committed

Commits on Aug 4, 2025

get_recent_video_capture tuple
Barbara Miller
committed

Commits on Aug 13, 2025

mv video data code to new file
Barbara Miller
committed

Commits on Aug 14, 2025

fix imports post-video_data move
Barbara Miller
committed
formatting fix
Barbara Miller
committed
import video_data.py when valid VIDEO_DATA_SOURCE
Barbara Miller
committed

Commits on Aug 15, 2025

datetime.now(datetime.timezone.utc)
Barbara Miller
committed
ruff'd
Barbara Miller
committed
recent_video_capture_exists
Barbara Miller
committed
predup check for all urls
Barbara Miller
committed
skip Bool
Barbara Miller
committed

Commits on Aug 20, 2025

use video_timestamp, mostly
Barbara Miller
committed
if result_tuple[0]
Barbara Miller
committed

Commits on Aug 21, 2025

more better result handling for recent_video_capture_exists
Barbara Miller
committed

Commits on Aug 23, 2025

fix buglets
Barbara Miller
committed
ruff'd
Barbara Miller
committed

Commits on Aug 25, 2025

fix last buglet & tweak logging
Barbara Miller
committed
ruff'd (added final comma 8|)
Barbara Miller
committed

Commits on Aug 27, 2025

should_ytdlp predup only when worker.video_data
Barbara Miller
committed
if self._video_data
Barbara Miller
committed
self._video_data = None sometimes
Barbara Miller
committed

Commits on Oct 30, 2025

note places we might check for duplicate display_id
Barbara Miller
committed