Bugfix: path dependencies module resolution bug with local paths by daniellionel01 · Pull Request #4563 · gleam-lang/gleam

daniellionel01 · 2025-05-05T18:00:32Z

Code to fix #2278

I hope this is an acceptable solution, I've inspected closed PRs that didn't make it attempting to fix this issue (#2334 and #3398).

In summary this approach stores a hash of the manifest.toml of all dependencies that are a path dependencies in the build directory and rebuilds any packages if the manifest.toml should change.

Let me know if I've missed anything or if this approach does not scale well, thank you!

Also I did not want to clutter the codebase with it, but here is a script (thank you GPT) to test the scenario that produces the error mentioned in the linked issue. It assumes you're running it in the root of the gleam codebase, otherwise you can adjust the GLEAM_BIN path. Also this script probably only works on a mac or linux. Here is the script:

#!/bin/bash
set -e

# Use the locally built gleam executable
GLEAM_BIN="$PWD/target/debug/gleam"

# Ensure the local build exists
if [ ! -f "$GLEAM_BIN" ]; then
    echo "Error: Local Gleam build not found at $GLEAM_BIN"
    echo "Please run 'cargo build' first"
    exit 1
fi

echo "Using Gleam binary at: $GLEAM_BIN"

# Create a temporary directory for the test
TEST_DIR=$(mktemp -d)
echo "Creating test environment in $TEST_DIR"

# Create bar (dependency) project
echo "Creating bar project (the dependency)"
cd $TEST_DIR
mkdir -p bar
cd bar

# Initialize bar project
cat > gleam.toml << EOF
name = "bar"
version = "0.1.0"

[dependencies]
gleam_stdlib = "~> 0.29"
EOF

# Create bar module
mkdir -p src
cat > src/bar.gleam << EOF
import gleam/io

pub fn say_hello() {
  io.println("Hello from bar!")
}
EOF

# Create foo (root) project
echo "Creating foo project (the root project)"
cd $TEST_DIR
mkdir -p foo
cd foo

# Initialize foo project with bar as path dependency
cat > gleam.toml << EOF
name = "foo"
version = "0.1.0"

[dependencies]
bar = { path = "../bar" }
EOF

# Create foo module that imports bar
mkdir -p src
cat > src/foo.gleam << EOF
import bar

pub fn main() {
  bar.say_hello()
}
EOF

# First build of foo
echo "Building foo for the first time..."
cd $TEST_DIR/foo
$GLEAM_BIN build

# Add simplifile dependency to bar
echo "Adding simplifile dependency to bar..."
cd $TEST_DIR/bar
$GLEAM_BIN add simplifile

# Update bar module to use simplifile
cat > src/bar.gleam << EOF
import simplifile
import gleam/io

pub fn say_hello() {
  io.println("Hello from bar!")
}
EOF

# Build foo again - this should now succeed with our fix
echo "Building foo again after adding simplifile to bar..."
cd $TEST_DIR/foo
$GLEAM_BIN build

if [ $? -eq 0 ]; then
  echo -e "\n\033[0;32mSUCCESS: The fix works! Foo successfully built after bar added simplifile dependency.\033[0m"
else
  echo -e "\n\033[0;31mFAILURE: The fix didn't work. Foo failed to build after bar added simplifile dependency.\033[0m"
  exit 1
fi

# Clean up
echo -e "\nTest directory: $TEST_DIR"
echo "You can safely remove this directory when done with: rm -rf $TEST_DIR"
echo "Or inspect it to see the project structure."

lpil

Sorry this took so long for me to review! I missed it somehow, and I've only just seen it now.

Thank you for this, I've left some comments inline. Tests will need to be added too.

lpil · 2025-06-25T15:26:28Z

+    // which indicates that their dependencies have changed
+    for (key, requirement2) in requirements2 {
+        if let Requirement::Path { path } = requirement2 {
+            let dep_manifest_path = root_path.join(path).join("manifest.toml");


All path manipulation must be done in the ProjectPaths class, no other code is permitted to use .join etc.

lpil · 2025-06-25T15:26:53Z

+
+    // We used to always force clean artifacts for path dependencies,
+    // but now we use the gleam.toml hash tracking in is_same_requirements
+    // to determine when path dependency dependencies have changed


What does this comment mean?

caught my GPT generated snippet. Sorry, removing it.

lpil · 2025-06-25T15:31:17Z


+    // For path dependencies, we need to check if their manifest.toml files have changed,
+    // which indicates that their dependencies have changed
+    for (key, requirement2) in requirements2 {


This equality checking logic should be in the same_requirements function as that's the one that checks equality, while this one iterates over the collection.

yeah, I removed the comment

lpil · 2025-06-25T15:33:25Z

+                }
+            };
+
+            let mut hasher = std::hash::DefaultHasher::new();


We use xxhash_rust::xxh3::xxh3_64 for fingerprinting, not the default hasher.

lpil · 2025-06-25T15:33:52Z

+                    tracing::debug!("cannot_read_path_dependency_manifest_forcing_rebuild");
+                    return Ok(false);
+                }
+            };


We check mtimes before we hash, to avoid extra work, so let's add that here.

lpil · 2025-06-25T15:34:36Z

+            // If cached hash file doesn't exist, this is the first time we're checking this dependency
+            if !cached_hash_path.exists() {
+                // Save the current hash for future comparisons
+                if let Err(e) = fs::write(&cached_hash_path, &current_hash) {


The errors in this code are being silently discarded! They must always be passed back up.

lpil · 2025-06-25T15:36:40Z

+
+            if !dep_manifest_path.exists() {
+                tracing::debug!("path_dependency_manifest_not_found_forcing_rebuild");
+                return Ok(false);


This would result in rebuilding every single time when there's no manifest for the path dep!

Sorry! continue-ing instead now

lpil · 2025-06-25T15:39:40Z

        }
    }

+    // For path dependencies, we need to check if their manifest.toml files have changed,


The name for this function doesn't fit what it does any more- even if the requirements are the same the function can return false in the instance that the definition of one of the required deps has changed.

The functions have also been made effectful and to depend on file system state, so that would need to be communicated clearly in the name and arguments too.

These functions that check the requirements are the same could go unchanged (which would still fit the "is same requirements" name) and a new function could be created to check just the path deps have not changed their deps.

pushed an update!

daniellionel01 · 2025-06-26T02:45:19Z

@lpil Addressed all comments!

Also added tests, that you can checkout.

Thank you for the input!

giacomocavalieri · 2026-02-13T15:10:44Z

Hello! If this is ready please remember to un-draft it, also would you mind rebasing on main? It seems there's some conflicts

daniellionel01 · 2026-02-13T15:51:06Z

Hello! If this is ready please remember to un-draft it, also would you mind rebasing on main? It seems there's some conflicts

i'll get on rebasing!

daniellionel01 · 2026-02-13T19:13:47Z

@lpil @giacomocavalieri ready to merge from my end!

After rebasing onto upstream/main, the dependency resolution logic was refactored into DependencyManager. This commit adapts the fix to work with the new architecture by: - Making is_same_requirements and are_path_dependency_manifests_unchanged pub(crate) so they can be used from dependency_manager.rs - Adding the path dependency manifest check to DependencyManager::resolve_versions - Updating tests to use the renamed function

The hashbrown version change (0.15.3 -> 0.15.5) altered HashMap iteration order, causing dependency resolution tests to fail. Restored upstream's Cargo.lock and rebuilt to include our dependencies.

lpil

Thank you!!!

lpil reviewed Jun 25, 2025

View reviewed changes

lpil marked this pull request as draft June 25, 2025 15:42

daniellionel01 force-pushed the bugfix/local-module-resolution branch from 358a62f to a2c41e2 Compare February 13, 2026 16:23

daniellionel01 marked this pull request as ready for review February 13, 2026 19:13

daniellionel01 added 16 commits February 26, 2026 10:56

create task file to track documentation

b376b27

bugfix: path dependencies module resolution bug with local paths

62b05c8

rm comment

57f302b

rm comment

790b381

install xxhash-rust in compiler-cli & update hash function

d630506

avoid extra work (hashing) by checking mtimes

c736912

make sure error is passed back up

1c2145c

fix: avoid rebuilding when manifest is missing

81bb8a6

add tests for outdated manifest checks

f6393ad

remove .join code outside of ProjectPaths

03a5858

signal effectfulness of function clearly

67243dd

fix: restore upstream Cargo.lock hashbrown versions

0939f27

The hashbrown version change (0.15.3 -> 0.15.5) altered HashMap iteration order, causing dependency resolution tests to fail. Restored upstream's Cargo.lock and rebuilt to include our dependencies.

refactor: rename function and remove unused param

6680187

cargo fmt

d351238

refactor: rename function and remove unused param

efecdb7

lpil force-pushed the bugfix/local-module-resolution branch from 945a8cd to efecdb7 Compare February 26, 2026 10:57

lpil added 2 commits February 26, 2026 12:00

Use configs rather than manifests

96cc027

Remove unused dep

fd7213e

giacomocavalieri approved these changes Feb 26, 2026

View reviewed changes

lpil approved these changes Feb 26, 2026

View reviewed changes

lpil enabled auto-merge (rebase) February 26, 2026 13:04

lpil merged commit f12c1ea into gleam-lang:main Feb 26, 2026
12 checks passed

Uh oh!

Uh oh!

Conversation

daniellionel01 commented May 5, 2025

Uh oh!

lpil left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniellionel01 Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniellionel01 commented Jun 26, 2025

Uh oh!

giacomocavalieri commented Feb 13, 2026

Uh oh!

daniellionel01 commented Feb 13, 2026

Uh oh!

daniellionel01 commented Feb 13, 2026

Uh oh!

lpil left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

daniellionel01 Jun 26, 2025 •

edited

Loading