Updates

Consolidate type checking into root tsconfig

- Add glob as explicit dependency (was transitive only)
- Remove website/tsconfig.json; root tsconfig covers it via the
  website/.astro/types.d.ts include which provides astro/client types
  (import.meta.glob etc.)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add vite/client reference to fix CI typecheck

website/.astro/types.d.ts is Astro-generated and absent in CI,
so ImportMeta.glob was untyped. Adding the reference to global.d.ts
ensures vite types are always loaded.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Taxonomy filters

Extract shared generate script and remove redundant buildonly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Updates

Couple more filters

Updates

Updates

Updates

Shell script hardening (#95)

* Website code quality fixes

- Fix pushState → replaceState in useSearchFilter (search was adding history entries per keystroke)
- Add res.ok checks in useSearchIndex and useTaxonomyFilter fetchers
- Fix non-null assertion on nullable filter param in useCategoryFilter
- Remove dead typeof window checks in useCategoryFilter, useColumnVisibility, useTableSort
- Consolidate duplicate statusOrder keys to lowercase, normalize lookup with .toLowerCase()
- Fix XSS in recently-updated renderTable: replace innerHTML string interpolation with DOM APIs
- Sync HubEntry interface in recently-updated script block (add missing createdTimestamp)
- Replace as any cast in blog/index.astro with proper AstroComponentFactory typing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Harden shell scripts: set -euo pipefail, xxhash, safe var refs

- Add set -euo pipefail to all scripts that were missing it
- Add set -o pipefail inside exported GFF-processing functions so
  pipelines fail properly in parallel subshells
- Fix bare $REPROCESS/$REDOWNLOAD/$REPROCESS_TRIX refs (need ${VAR:-}
  under set -u); fix ${1} -> ${1:-} in cleanupStaleGff.sh
- Fix /tmp/datasets_err hardcoded path in genark2jbrowse/make.sh
  (use mktemp instead)
- Fix ! $DRY_RUN && log patterns in cleanupStaleGff.sh that would
  trip set -e when DRY_RUN=true
- Remove unnecessary || true from textIndexGoldenPath.sh parallel call
  (process_assembly already handles failures gracefully)
- Switch .trackdb_hash and fileListing.txt from stat file-size to
  xxhsum (requires: sudo apt install xxhash); note existing .trackdb_hash
  files will all be treated as changed on first run

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Restore || true for textIndexGoldenPath parallel call

text-index failures are expected (track IDs can be missing from the
index), so the parallel run should never abort the parent pipeline.
Add --halt never to also be explicit about the intent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix createChainTrackPifs failing on assemblies with no liftOver chains

wget exit code 7 (protocol error) when an assembly has no liftOver
directory was propagating through the pipeline after makePifs.sh got
set -euo pipefail. Fixes:

- extract_file_urls: capture wget output via $() so a failure returns
  empty string (exit 0) rather than crashing the pipeline
- "no chain files found" is now a soft skip (log_info + return 0)
  rather than log_error — many assemblies legitimately have no chains
- makePifs.sh: add --halt never + || true to both parallel calls as a
  backstop for any other per-assembly failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Skip PIF generation for already-processed assemblies

Use a .checked stamp file in each liftOver/vs dir so subsequent runs
skip the HTTP directory fetch entirely. Pre-filter assembly lists in
make.sh/makePifs.sh before parallel to avoid forking processes for
already-stamped assemblies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Format

* Incremental file listing with XXH3: ~3x faster hashing

- Add make_file_listing() to common.sh: uses find -newer to skip
  unchanged files, merges with existing listing, drops deleted entries
- Switch from default XXH64 to XXH3 (-H3): 62 GB/s vs 19 GB/s on this
  machine; algo_tag header triggers full re-hash on algorithm change
- Simplify getFileListing.sh and ucsc2jbrowse/make.sh to single-line
  calls; remove duplicated inline logic
- Delete stale fileListing.txt files (XXH64 format) so next run
  rebuilds cleanly with XXH3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Dry run output: remove bogus over.chain.gz liftOver tracks, reorder aggregateTextSearchAdapters

- ~1700 liftOver synteny tracks with queryAssembly "over.chain.gz" removed;
  these were generated erroneously when assemblies had no liftOver directory
  (fixed by createChainTrackPifs patch in prior commit)
- aggregateTextSearchAdapters moved before plugins in minimal configs
  (cosmetic reordering, no content change)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update

* Fix defaultSession missing from minimal configs: run generateDefaultSessions before createMinimalConfigs

createMinimalConfigs reads config.json and passes all top-level keys
through, but defaultSession wasn't present yet because generateDefaultSessions
ran after it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add set -euo pipefail to remaining ucsc2jbrowse scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Move files around

Add shfmt formatting and fix parallel pipeline failures

- Format all shell scripts with shfmt v3.13.1 (2-space indent)
- Add shfmt to format and lint:sh scripts in package.json
- Install shfmt v3.13.1 in CI lint workflow
- Add || true to parallel calls for per-assembly operations so individual
  failures (GFF processing, add-track, text-index, metadata) don't abort
  the whole pipeline
- Fix addNcbiGffAndTextIndex.sh to handle add-track failures with a warning
  and return, matching the inline version in make.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Run astro sync before typecheck in CI to generate astro:content types

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Migrate to Astro 6 Content Layer API and add astro check to CI

- Move content config to src/content.config.ts with glob loader
- Replace deprecated post.render() with render(post) from astro:content
- Import z from astro/zod instead of deprecated astro:content export
- Add astro check step to CI to type-check .astro files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Lints

Actionsup

Bump deps

Updates

Author: cmdcolin

.github/workflows/lint.yml

@@ -10,24 +10,35 @@ jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
 
-      - uses: pnpm/action-setup@v2
+      - uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v5.0.0
         with:
-          version: 8
+          version: 10
 
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
         with:
-          node-version: '20'
+          node-version: '24'
           cache: 'pnpm'
 
       - run: pnpm install
 
+      - name: Astro sync
+        run: cd website && pnpm astro sync
+
+      - name: Astro check
+        run: cd website && pnpm astro check
+
       - name: TypeScript type check
         run: pnpm typecheck
 
       - name: ESLint
         run: pnpm lint
 
+      - name: Install shfmt
+        run: |
+          curl -sSL "https://github.com/mvdan/sh/releases/download/v3.13.1/shfmt_v3.13.1_linux_amd64" -o /usr/local/bin/shfmt
+          chmod +x /usr/local/bin/shfmt
+
       - name: ShellCheck
         run: pnpm lint:sh

.prettierrc.json

@@ -9,6 +9,7 @@
     }
   ],
   "semi": false,
+  "singleAttributePerLine": true,
   "singleQuote": true,
   "trailingComma": "all",
   "arrowParens": "avoid",

CLAUDE.md

@@ -0,0 +1,24 @@
+# jb2hubs
+
+Monorepo that converts UCSC GenArk and UCSC browser hubs into JBrowse 2 configs,
+and serves them via a static website.
+
+## Packages
+
+- `website/` — Astro + React static site (pages: search, recently-updated,
+  accession, taxonomy, hubs, synteny, etc.)
+- `genark2jbrowse/` — scripts + TS that process GenArk hubs into JBrowse configs
+- `ucsc2jbrowse/` — scripts + TS that convert UCSC track hubs into JBrowse
+  configs
+- `hubtools/` — shared TS library used by the converter packages
+
+## Key website internals
+
+- `src/components/SearchPage.tsx` — client-side search over
+  `public/searchIndex.json`
+- `src/pages/recently-updated.astro` — server-rendered table with category
+  dropdown filter
+- `src/hooks/useSearchIndex.ts` — SWR fetch of the search index;
+  `IndexEntry = [accession, commonName, scientificName, assemblyName, assemblyStatus, source, taxonId]`
+- `src/recentlyUpdated.json` — build-time generated data for recently-updated
+  page

aws/config-merger/src/index.ts

@@ -1,7 +1,10 @@
-import { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from 'aws-lambda'
-
 import { mergeConfigs } from './merger.ts'
-import { JBrowseConfig, MergeOptions } from './types.ts'
+
+import type { JBrowseConfig, MergeOptions } from './types.ts'
+import type {
+  APIGatewayProxyEventV2,
+  APIGatewayProxyResultV2,
+} from 'aws-lambda'
 
 function addRelativeUris(config: unknown, baseUri: string) {
   if (typeof config === 'object' && config !== null) {

aws/config-merger/src/merger.ts

@@ -1,4 +1,4 @@
-import {
+import type {
   AggregateTextSearchAdapter,
   Assembly,
   JBrowseConfig,

aws/config-merger/test/merger.test.ts

@@ -2,7 +2,8 @@ import { describe, expect, it } from 'vitest'
 
 import { addRelativeUris, idToConfigUrl } from '../src/index.ts'
 import { mergeConfigs } from '../src/merger.ts'
-import { Assembly, JBrowseConfig, SyntenyTrack } from '../src/types.ts'
+
+import type { Assembly, JBrowseConfig, SyntenyTrack } from '../src/types.ts'
 
 function makeAssembly(name: string, displayName?: string): Assembly {
   return {

common.sh

@@ -31,3 +31,45 @@ ensure_dir() {
   mkdir -p "$1"
 }
 export -f ensure_dir
+
+# Incrementally updates a hash listing file using XXH3.
+# Only re-hashes files newer than the listing; handles additions and deletions.
+# Usage: make_file_listing <listing> <find_dir> [extra_find_args...]
+make_file_listing() {
+  local listing="$1" find_dir="$2"
+  shift 2
+  local extra_args=("$@")
+  local algo="-H3"
+  local algo_tag="# algo=xxh3"
+  local tmp_new tmp_cur
+  tmp_new=$(mktemp)
+  tmp_cur=$(mktemp)
+
+  if [[ ! -f "$listing" ]] || ! head -1 "$listing" | grep -qF "$algo_tag"; then
+    find "$find_dir" -type f "${extra_args[@]}" -exec xxhsum "$algo" {} + | sort -k2,2 >"${listing}.tmp"
+    {
+      echo "$algo_tag"
+      cat "${listing}.tmp"
+    } >"$listing"
+    rm -f "${listing}.tmp" "$tmp_new" "$tmp_cur"
+    return 0
+  fi
+
+  find "$find_dir" -type f "${extra_args[@]}" -newer "$listing" -exec xxhsum "$algo" {} + >"$tmp_new"
+  find "$find_dir" -type f "${extra_args[@]}" | sort >"$tmp_cur"
+
+  tail -n +2 "$listing" |
+    awk 'NR==FNR{skip[$2]=1; next} !($2 in skip)' "$tmp_new" - |
+    awk 'NR==FNR{exists[$1]=1; next} ($2 in exists)' "$tmp_cur" - |
+    {
+      cat
+      cat "$tmp_new"
+    } |
+    sort -k2,2 >"${listing}.tmp"
+  {
+    echo "$algo_tag"
+    cat "${listing}.tmp"
+  } >"$listing"
+  rm -f "${listing}.tmp" "$tmp_new" "$tmp_cur"
+}
+export -f make_file_listing

eslint.config.mjs

@@ -19,7 +19,7 @@ export default tseslint.config(
     languageOptions: {
       parserOptions: {
         projectService: {
-          allowDefaultProject: ['*.js'],
+          allowDefaultProject: ['*.js', '*.mjs', 'website/*.mjs'],
           defaultProject: './tsconfig.json',
         },
         tsconfigRootDir: import.meta.dirname,

genark2jbrowse/addNcbiGffAndTextIndex.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -euo pipefail
+
 source "$(dirname "$0")/common.sh"
 
 # Define function to add a GFF track to a JBrowse 2 assembly and create a text index.
@@ -26,9 +28,12 @@ add_track_and_text_index() {
     return 0
   fi
 
-  jbrowse add-track --force "$gff_file_path" --out "$hub_dir" --load copy --indexFile "${gff_file_path}".csi --trackId "${accession}-ncbiGff" --name "NCBI RefSeq - RefSeq All (GFF)" --category "Genes and Gene Predictions" >/dev/null
+  if ! jbrowse add-track --force "$gff_file_path" --out "$hub_dir" --load copy --indexFile "${gff_file_path}".csi --trackId "${accession}-ncbiGff" --name "NCBI RefSeq - RefSeq All (GFF)" --category "Genes and Gene Predictions" >/dev/null; then
+    echo "Warning: add-track failed for $accession" >&2
+    return
+  fi
   # Check if trix folder exists
-  if [ -d "$hub_dir/trix" ] && [ -z "$REDOWNLOAD" ] && [ -z "$REPROCESS" ] && [ -z "$REPROCESS_TRIX" ]; then
+  if [ -d "$hub_dir/trix" ] && [ -z "${REDOWNLOAD:-}" ] && [ -z "${REPROCESS:-}" ] && [ -z "${REPROCESS_TRIX:-}" ]; then
     add_trix_adapter "$accession" "$config_file"
   else
     echo "Trix folder does not exist for $accession, running jbrowse text-index"
@@ -40,4 +45,4 @@ add_track_and_text_index() {
 # Export function for use with GNU Parallel
 export -f add_track_and_text_index
 
-find bgz -name "*.gz" | parallel -j16 $PARALLEL_OPTS add_track_and_text_index
+find bgz -name "*.gz" | parallel -j16 $PARALLEL_OPTS add_track_and_text_index || true

genark2jbrowse/cleanupStaleGff.sh

@@ -11,18 +11,20 @@
 #   ./cleanupStaleGff.sh --exec # actually delete the files
 #
 
+set -euo pipefail
+
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 ALL_JSON="$SCRIPT_DIR/processedHubJson/all.json"
 LOG_FILE="$SCRIPT_DIR/CLEANED.md"
 DRY_RUN=true
 
-if [ "${1}" = "--exec" ]; then
+if [ "${1:-}" = "--exec" ]; then
   DRY_RUN=false
 fi
 
 log() {
   if ! $DRY_RUN; then
-    echo "$1" >> "$LOG_FILE"
+    echo "$1" >>"$LOG_FILE"
   fi
 }
 
@@ -41,7 +43,7 @@ delete_file() {
 
 # Build a temp file of known GFF basenames for fast lookup
 known_basenames_file=$(mktemp)
-jq -r '.[].ncbiGff | select(. != null) | split("/") | last' "$ALL_JSON" > "$known_basenames_file"
+jq -r '.[].ncbiGff | select(. != null) | split("/") | last' "$ALL_JSON" >"$known_basenames_file"
 
 cleanup() { rm -f "$known_basenames_file"; }
 trap cleanup EXIT
@@ -51,13 +53,13 @@ is_known() {
 }
 
 if ! $DRY_RUN; then
-  echo "# Cleanup log ($(date -u '+%Y-%m-%d %H:%M UTC'))" > "$LOG_FILE"
-  echo "" >> "$LOG_FILE"
+  echo "# Cleanup log ($(date -u '+%Y-%m-%d %H:%M UTC'))" >"$LOG_FILE"
+  echo "" >>"$LOG_FILE"
 fi
 
 # Remove leftover uncompressed .gff files in bgz/
 echo "=== Leftover uncompressed .gff files in bgz/ ==="
-! $DRY_RUN && log "## Leftover uncompressed .gff files in bgz/"
+if ! $DRY_RUN; then log "## Leftover uncompressed .gff files in bgz/"; fi
 for f in "$SCRIPT_DIR/bgz/"*.gff; do
   [ -f "$f" ] || continue
   delete_file "$f"
@@ -66,7 +68,10 @@ done
 # Remove .csi files in bgz/ with no corresponding .gz
 echo ""
 echo "=== Orphaned .csi files in bgz/ ==="
-! $DRY_RUN && log "" && log "## Orphaned .csi files in bgz/"
+if ! $DRY_RUN; then
+  log ""
+  log "## Orphaned .csi files in bgz/"
+fi
 for f in "$SCRIPT_DIR/bgz/"GC[FA]_*.gz.csi; do
   [ -f "$f" ] || continue
   if [ ! -f "${f%.csi}" ]; then
@@ -79,7 +84,10 @@ done
 for dir in gff bgz; do
   echo ""
   echo "=== GFF files not in listing ($dir/) ==="
-  ! $DRY_RUN && log "" && log "## GFF files not in listing ($dir/)"
+  if ! $DRY_RUN; then
+    log ""
+    log "## GFF files not in listing ($dir/)"
+  fi
 
   for f in "$SCRIPT_DIR/$dir/"GC[FA]_*.gz; do
     [ -f "$f" ] || continue