[tests, docker, docs, scripts, ci] chore: remove internal-only assets, add sensitive scan - #174
Open
NeosZhang wants to merge 2 commits into
Open
[tests, docker, docs, scripts, ci] chore: remove internal-only assets, add sensitive scan#174NeosZhang wants to merge 2 commits into
NeosZhang wants to merge 2 commits into
Conversation
github-actions
Bot
added
bug
nullnonenilNULL
changed the title
NeosZhang
force-pushed
the
sanitize/internal-refs
branch
2 times, most recently
from
August 20, 2026 12:44
45e43a6 to
ed0af71
Compare
…, add sensitive scan Drop internal build/CI assets and replace internal-infrastructure defaults with neutral placeholders so a fresh clone is self-contained for external users, and add a scanner that keeps internal-only strings from coming back. Packaging / infra: - remove the internal XPU docker assets and the P800 install tutorial (both the English and Chinese copies), and drop the toctree entries and cross-links that pointed at it - ignore local AI-assistant rule directories Sensitive-information scan: - add ci/sensitive_scan.py and ci/sensitive_rules.py, which flag internal hostnames and package mirrors, internal registries, private object-storage locations, corporate emails, developer home paths and credential material; findings are error- or warn-severity, with inline `# sensitive-scan: allow[rule-id]` and an ALLOWLIST for reviewed exceptions - add skills/sensitive-scan/SKILL.md documenting how to run and extend it - allowlist the `bos:/$BOS_BUCKET/...` uploads in the workflows: the bucket name comes from a repository variable, not from the source Examples / configs: - default the Kimi NVFP4 blobfile install to the public PyPI index, with BLOBFILE_PIP_INDEX / BLOBFILE_PIP_TRUSTED_HOST as opt-in mirror overrides - replace internal cluster paths and dataset locations in examples, configs and tests/embodied/config/prepare.sh with placeholders LLM/VLM E2E suite: - remove prepare_env.sh / download_datasets.sh, which bootstrapped models and datasets from internal object storage, plus their invocations in main_start.sh and pipeline_runner.sh - drop the --ak / --sk credential passthrough and --skip_env - remove per-model bos_path entries from the test configs - remove the internal docker build assets and unreleased-chip baselines - remove the internal pipeline launcher (ipipe_start.sh), its PyTorchJob yaml templates, and the internal notification helpers (utils/send_msg.sh, utils/HiRobotApi.py), along with the k8s/pipeline plumbing they required in common/common.sh and main.py
NeosZhang
force-pushed
the
sanitize/internal-refs
branch
from
August 20, 2026 13:18
ed0af71 to
d24daa1
Compare
NeosZhang
changed the title
Zachary-wW
force-pushed
the
sanitize/internal-refs
branch
from
August 21, 2026 08:23
91b1043 to
fe2754a
Compare
Zachary-wW
force-pushed
the
sanitize/internal-refs
branch
from
August 21, 2026 08:28
fe2754a to
2d86230
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
master; no internal history is carried over.Changes
Packaging / infra
--trusted-hostfromops/requirements.txt.docker/Dockerfile.xpu,docker/xpu_packages.json,docker/fix_xpytorch_hook.sh,docker/__init__.py) and the P800 install tutorial..gitignore.Examples / configs
examples/kimi_k2.x/kimi_k2.{5,6}/checkpoint_convert/nvfp4_ptq/install_nvfp4_modelopt_deps.sh:blobfilenow installs from the default PyPI index.BLOBFILE_PIP_INDEX/BLOBFILE_PIP_TRUSTED_HOSTare optional overrides for users behind a local mirror; the plain-HTTP mirror is no longer a baked-in default.examples/embodied/xvla/xvla_soft_fold/metadata.json,examples/internvl3.5/finetuning/,configs/models/deepseek4/,tools/data_preprocess/vlm/offline_packing/, andtests/embodied/config/prepare.sh.LLM/VLM E2E suite (
tests/llm_vlm/)prepare_env.shanddownload_datasets.sh, which bootstrapped models and datasets from internal object storage, plus their invocations inmain_start.shandpipeline_runner.sh.--ak/--skcredential passthrough and--skip_envfrompipeline_runner.sh(unused once the bootstrap is gone).bos_pathentries from the default and optional configs.README.mdnow states that datasets, HF base models, and converted checkpoints must be provisioned before running the suite.Test Plan
bash -npasses on every modified shell script.install_nvfp4_modelopt_deps.shverified for all three cases:pip install blobfile -qpip install blobfile -q -i <index>pip install blobfile -q -i <index> --trusted-host <host>The array is seeded with
-qso it is never empty, which would fail underset -uon bash 4.2.