(v0.8.26) Contract code changes with additional files in compilation

[kuzdogan]

Here I am with another instance of #14494 and #14250.

Submitting this issue because these are supposed to be fixed in the latest versions but I encountered this originally in v0.8.26, and was able to also reproduce in v0.8.28

JSON Inputs

There are two instances (ie. std JSONs) of the same contract. One with extra files and one with only the files used by the compilation.

Below are the outputs with 0.8.26+commit.8a97fa7a. I haven't noticed any differences across binaries solc-js vs Darwin

• etherscan-response-input.json (Larger JSON): This is the full JSON with extra files, called "etherscan-response" because this is from Etherscan and its the content of the response

  • 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json

• sourcify-etherscan-import-input.json (Minimal JSON): This is the minimal JSON because we first import the file from Etherscan, generate the metadata by compiling the Larger JSON, and compile with the JSON based on this metadata. Since metadata.json generated strips away the source files not used by this contract, this creats a minimal JSON (expectedly).

  • 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json

Input differences

Here's the output of the script that checks for source file differences between the two JSON inputs:

Script Output

$ node compare-sources.js 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json

Keys in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json but not in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json: [
  '@openzeppelin/contracts/access/AccessControl.sol',
  '@openzeppelin/contracts/access/Ownable.sol',
  '@openzeppelin/contracts/interfaces/IERC5267.sol',
  '@openzeppelin/contracts/proxy/beacon/BeaconProxy.sol',
  '@openzeppelin/contracts/proxy/beacon/IBeacon.sol',
  '@openzeppelin/contracts/proxy/beacon/UpgradeableBeacon.sol',
  '@openzeppelin/contracts/proxy/Clones.sol',
  '@openzeppelin/contracts/proxy/ERC1967/ERC1967Utils.sol',
  '@openzeppelin/contracts/proxy/Proxy.sol',
  '@openzeppelin/contracts/token/ERC20/ERC20.sol',
  '@openzeppelin/contracts/token/ERC20/extensions/IERC20Permit.sol',
  '@openzeppelin/contracts/token/ERC20/utils/SafeERC20.sol',
  '@openzeppelin/contracts/token/ERC721/ERC721.sol',
  '@openzeppelin/contracts/token/ERC721/extensions/ERC721Burnable.sol',
  '@openzeppelin/contracts/token/ERC721/extensions/IERC721Metadata.sol',
  '@openzeppelin/contracts/token/ERC721/IERC721.sol',
  '@openzeppelin/contracts/token/ERC721/IERC721Receiver.sol',
  '@openzeppelin/contracts/utils/Address.sol',
  '@openzeppelin/contracts/utils/Context.sol',
  '@openzeppelin/contracts/utils/cryptography/EIP712.sol',
  '@openzeppelin/contracts/utils/cryptography/MessageHashUtils.sol',
  '@openzeppelin/contracts/utils/introspection/ERC165.sol',
  '@openzeppelin/contracts/utils/ShortStrings.sol',
  '@openzeppelin/contracts/utils/StorageSlot.sol',
  'contracts-exposed/LoanNFT/LoanXNFT.sol',
  'contracts-exposed/LoanPaymentGateway/ImplLoanPaymentGateway.sol',
  'contracts-exposed/LoanPaymentGateway/LoanPaymentGatewayFactory.sol',
  'contracts-exposed/LoanX/LoanX.sol',
  'contracts-exposed/mocks/MockedUSDC.sol',
  'contracts-exposed/mocks/MockedUSDT.sol',
  'contracts-exposed/SecurityToken/BeaconFactory/BeaconFactory.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/access/AccessControlModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/controllers/ERC20ControllerModule/ERC20ControllerModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/controllers/SignatureVerifier/SignatureVerifier.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20BaseModule/ERC20BaseModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20FrozenModule/ERC20FrozenModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20IssueModule/ERC20IssueModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/ERC20RedeemModule/ERC20RedeemModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/InfoModule/InfoModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/PauseModule/PauseModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/core/SnapshotModule/ERC20SnapshotModule.sol',
  'contracts-exposed/SecurityToken/SecurityToken/modules/SecurityTokenBase.sol',
  'contracts-exposed/SecurityToken/SecurityToken/SecurityTokenProxy.sol',
  'contracts-exposed/SecurityTokenPaymentGateway/BaseSecurityTokenPaymentGatewayFactory.sol',
  'contracts-exposed/SecurityTokenPaymentGateway/ImplementationEU.sol',
  'contracts-exposed/SecurityTokenPaymentGateway/ImplementationUS.sol',
  'contracts-exposed/SecurityTokenPaymentGateway/library/CloneWithStorage.sol',
  'contracts-exposed/SecurityTokenPaymentGateway/SecurityTokenPaymentGatewayFactory.sol',
  'contracts/LoanNFT/interfaces/ILoanXNFT.sol',
  'contracts/LoanNFT/LoanXNFT.sol',
  'contracts/LoanPaymentGateway/ImplLoanPaymentGateway.sol',
  'contracts/LoanPaymentGateway/interfaces/IImplLoanPaymentGateway.sol',
  'contracts/LoanPaymentGateway/interfaces/ILoanPaymentGatewayFactory.sol',
  'contracts/LoanPaymentGateway/interfaces/ILoanPaymentGatewayFactoryErrors.sol',
  'contracts/LoanPaymentGateway/LoanPaymentGatewayFactory.sol',
  'contracts/LoanX/interfaces/ILoanX.sol',
  'contracts/LoanX/interfaces/ILoanXErrors.sol',
  'contracts/LoanX/interfaces/ILoanXStructs.sol',
  'contracts/LoanX/LoanX.sol',
  'contracts/mocks/MockedUSDC.sol',
  'contracts/mocks/MockedUSDT.sol',
  'contracts/SecurityToken/BeaconFactory/BeaconFactory.sol',
  'contracts/SecurityToken/BeaconFactory/interfaces/IBeaconFactory.sol',
  'contracts/SecurityToken/BeaconFactory/interfaces/IBeaconFactoryErrors.sol',
  'contracts/SecurityTokenPaymentGateway/BaseSecurityTokenPaymentGatewayFactory.sol',
  'contracts/SecurityTokenPaymentGateway/ImplementationEU.sol',
  'contracts/SecurityTokenPaymentGateway/ImplementationUS.sol',
  'contracts/SecurityTokenPaymentGateway/interfaces/IBaseSecurityTokenPaymentGatewayFactory.sol',
  'contracts/SecurityTokenPaymentGateway/interfaces/IBaseSecurityTokenPaymentGatewayFactoryErrors.sol',
  'contracts/SecurityTokenPaymentGateway/interfaces/IImplementationEU.sol',
  'contracts/SecurityTokenPaymentGateway/interfaces/IImplementationUS.sol',
  'contracts/SecurityTokenPaymentGateway/interfaces/ISecurityTokenPaymentGatewayFactory.sol',
  'contracts/SecurityTokenPaymentGateway/library/CloneWithStorage.sol',
  'contracts/SecurityTokenPaymentGateway/SecurityTokenPaymentGatewayFactory.sol'
]
Keys in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-input.json but not in 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input.json: []

The JS script

const fs = require("fs");

// Function to read and parse JSON files
function readJSONFile(filePath) {
  try {
    const data = fs.readFileSync(filePath, "utf8");
    return JSON.parse(data);
  } catch (error) {
    console.error(`Error reading or parsing file ${filePath}:`, error);
    process.exit(1);
  }
}

// Get file paths from command-line arguments
const [file1Path, file2Path] = process.argv.slice(2);

if (!file1Path || !file2Path) {
  console.error("Please provide paths for both files as arguments.");
  process.exit(1);
}

// Read and parse the JSON files
const file1 = readJSONFile(file1Path);
const file2 = readJSONFile(file2Path);

// Extract the keys of the sources objects
const file1Keys = Object.keys(file1.sources);
const file2Keys = Object.keys(file2.sources);

// Find keys that are in file1 but not in file2
const inFile1NotInFile2 = file1Keys.filter((key) => !file2Keys.includes(key));

// Find keys that are in file2 but not in file1
const inFile2NotInFile1 = file2Keys.filter((key) => !file1Keys.includes(key));

// Output the results
console.log(`Keys in ${file1Path} but not in ${file2Path}:`, inFile1NotInFile2);
console.log(`Keys in ${file2Path} but not in ${file1Path}:`, inFile2NotInFile1);

Bytecode differences

Compiled runtime bytecode of the Larger JSON:
11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-runtime-bytecode.txt

Compiled runtime bytecode of the Minimal JSON:
11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-runtime-bytecode.txt

Even though they are mostly identical, the bytecodes are also different in lengths

• Larger JSON bytecode: 30000 characters
• Minimal JSON bytecode: 30016 characters

Taking the bytecode diff:

git diff --word-diff --word-diff-regex=. 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-runtime-bytecode.txt 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-import-from-etherscan-output-runtime-bytecode.txt

Some screenshots from the diff.

Middle:

Only the last part:

In the Middle example, you can see the pattern 3 being replaced by b

In the Last Part, there are longer additions in green (hence the difference in lengths). You can see the metadata hashes are identical.

v0.8.28

I also reproduced this with 0.8.28. The Minimal JSON compiles directly with 0.8.28, although the Larger JSON has strict versions pragma solidity 0.8.26; so I need to manually set them to pragma solidity 0.8.28;

Larger JSON with replaced 0.8.28:
11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-input-0.8.28.json

Minimal JSON is identical

Larger JSON 0.8.28 runtime bytecode:
11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-0.8.28-runtime-bytecode.txt

Minimal JSON 0.8.28 runtime bytecode:
11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt

Diff command

git diff --word-diff --word-diff-regex=. 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-etherscan-response-output-0.8.28-runtime-bytecode.txt 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt

Similar noticable patterns:

Middle:

Ending:

The bytecode lengths are same as before (30000 vs 30016 characters). The bytecode between same JSONs but different versions ("Larger JSON 0.8.26" vs. "Larger JSON 0.8.28") are different as expected. But from what I can see the shifts are the same, ie. the diff screenshots above and here are almost identical.

Notice the metadata hashes are again identical. One might have expected the metadata hash to change since we touched pragma statements, however we only touched the files that are not supposed to be used in the compilation. Therefore the metadata hashes remain the same expectedly.

For convenience I'm also adding the command I used to extract the runtime bytecode from the JSON outputs:

cat 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28.json  | jq '.contracts.["contracts/SecurityToken/SecurityToken/SecurityTokenProxy.sol"].SecurityTokenProxy.evm.deployedBytecode.object' > 11155111-0xAD7714459F728E78279a77A019fE948A21480B83-sourcify-etherscan-import-output-0.8.28-runtime-bytecode.txt

[ekpyron]

@clonker Can you check if the ID changes would solve this case (if you still have a working branch)?

[clonker]

My working branch (yul_node_handles) is quite antiquated by now but it does solve this case. I'll try to push along the progress on this front this week a bit.