Skip to content

Bugfix: path dependencies module resolution bug with local paths #4563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

daniellionel01
Copy link

Code to fix #2278

I hope this is an acceptable solution, I've inspected closed PRs that didn't make it attempting to fix this issue (#2334 and #3398).

In summary this approach stores a hash of the manifest.toml of all dependencies that are a path dependencies in the build directory and rebuilds any packages if the manifest.toml should change.

Let me know if I've missed anything or if this approach does not scale well, thank you!

Also I did not want to clutter the codebase with it, but here is a script (thank you GPT) to test the scenario that produces the error mentioned in the linked issue. It assumes you're running it in the root of the gleam codebase, otherwise you can adjust the GLEAM_BIN path. Also this script probably only works on a mac or linux. Here is the script:

#!/bin/bash
set -e

# Use the locally built gleam executable
GLEAM_BIN="$PWD/target/debug/gleam"

# Ensure the local build exists
if [ ! -f "$GLEAM_BIN" ]; then
    echo "Error: Local Gleam build not found at $GLEAM_BIN"
    echo "Please run 'cargo build' first"
    exit 1
fi

echo "Using Gleam binary at: $GLEAM_BIN"

# Create a temporary directory for the test
TEST_DIR=$(mktemp -d)
echo "Creating test environment in $TEST_DIR"

# Create bar (dependency) project
echo "Creating bar project (the dependency)"
cd $TEST_DIR
mkdir -p bar
cd bar

# Initialize bar project
cat > gleam.toml << EOF
name = "bar"
version = "0.1.0"

[dependencies]
gleam_stdlib = "~> 0.29"
EOF

# Create bar module
mkdir -p src
cat > src/bar.gleam << EOF
import gleam/io

pub fn say_hello() {
  io.println("Hello from bar!")
}
EOF

# Create foo (root) project
echo "Creating foo project (the root project)"
cd $TEST_DIR
mkdir -p foo
cd foo

# Initialize foo project with bar as path dependency
cat > gleam.toml << EOF
name = "foo"
version = "0.1.0"

[dependencies]
bar = { path = "../bar" }
EOF

# Create foo module that imports bar
mkdir -p src
cat > src/foo.gleam << EOF
import bar

pub fn main() {
  bar.say_hello()
}
EOF

# First build of foo
echo "Building foo for the first time..."
cd $TEST_DIR/foo
$GLEAM_BIN build

# Add simplifile dependency to bar
echo "Adding simplifile dependency to bar..."
cd $TEST_DIR/bar
$GLEAM_BIN add simplifile

# Update bar module to use simplifile
cat > src/bar.gleam << EOF
import simplifile
import gleam/io

pub fn say_hello() {
  io.println("Hello from bar!")
}
EOF

# Build foo again - this should now succeed with our fix
echo "Building foo again after adding simplifile to bar..."
cd $TEST_DIR/foo
$GLEAM_BIN build

if [ $? -eq 0 ]; then
  echo -e "\n\033[0;32mSUCCESS: The fix works! Foo successfully built after bar added simplifile dependency.\033[0m"
else
  echo -e "\n\033[0;31mFAILURE: The fix didn't work. Foo failed to build after bar added simplifile dependency.\033[0m"
  exit 1
fi

# Clean up
echo -e "\nTest directory: $TEST_DIR"
echo "You can safely remove this directory when done with: rm -rf $TEST_DIR"
echo "Or inspect it to see the project structure."

Copy link
Member

@lpil lpil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry this took so long for me to review! I missed it somehow, and I've only just seen it now.

Thank you for this, I've left some comments inline. Tests will need to be added too.

// which indicates that their dependencies have changed
for (key, requirement2) in requirements2 {
if let Requirement::Path { path } = requirement2 {
let dep_manifest_path = root_path.join(path).join("manifest.toml");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All path manipulation must be done in the ProjectPaths class, no other code is permitted to use .join etc.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!


// We used to always force clean artifacts for path dependencies,
// but now we use the gleam.toml hash tracking in is_same_requirements
// to determine when path dependency dependencies have changed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this comment mean?

Copy link
Author

@daniellionel01 daniellionel01 Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

caught my GPT generated snippet. Sorry, removing it.

@@ -672,6 +672,66 @@ fn is_same_requirements(
}
}

// For path dependencies, we need to check if their manifest.toml files have changed,
// which indicates that their dependencies have changed
for (key, requirement2) in requirements2 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This equality checking logic should be in the same_requirements function as that's the one that checks equality, while this one iterates over the collection.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I removed the comment

}
};

let mut hasher = std::hash::DefaultHasher::new();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use xxhash_rust::xxh3::xxh3_64 for fingerprinting, not the default hasher.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

tracing::debug!("cannot_read_path_dependency_manifest_forcing_rebuild");
return Ok(false);
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We check mtimes before we hash, to avoid extra work, so let's add that here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

// If cached hash file doesn't exist, this is the first time we're checking this dependency
if !cached_hash_path.exists() {
// Save the current hash for future comparisons
if let Err(e) = fs::write(&cached_hash_path, &current_hash) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The errors in this code are being silently discarded! They must always be passed back up.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!


if !dep_manifest_path.exists() {
tracing::debug!("path_dependency_manifest_not_found_forcing_rebuild");
return Ok(false);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would result in rebuilding every single time when there's no manifest for the path dep!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry! continue-ing instead now

@@ -672,6 +672,66 @@ fn is_same_requirements(
}
}

// For path dependencies, we need to check if their manifest.toml files have changed,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name for this function doesn't fit what it does any more- even if the requirements are the same the function can return false in the instance that the definition of one of the required deps has changed.

The functions have also been made effectful and to depend on file system state, so that would need to be communicated clearly in the name and arguments too.

These functions that check the requirements are the same could go unchanged (which would still fit the "is same requirements" name) and a new function could be created to check just the path deps have not changed their deps.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pushed an update!

@lpil lpil marked this pull request as draft June 25, 2025 15:42
@daniellionel01
Copy link
Author

@lpil Addressed all comments!

Also added tests, that you can checkout.

Thank you for the input!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Path dependencies module resolution bug
2 participants