Escape absolute paths with Z: also in response files #96

marcoesposito1988 · 2023-11-27T08:31:56Z

Solves the issue discussed in #93

mstorsjo · 2023-12-10T21:48:08Z

Can @huangqinjin have a look at this?

huangqinjin · 2023-12-14T06:40:02Z

@marcoesposito1988 could you please provide some rsp files here for reference? You can pass -d keeprsp to ninja to prevent rsp files being removed .

marcoesposito1988 · 2023-12-15T23:08:49Z

@marcoesposito1988 could you please provide some rsp files here for reference? You can pass -d keeprsp to ninja to prevent rsp files being removed .

Here is an example:

-DBOOST_ALL_NO_LIB -DBOOST_UUID_FORCE_AUTO_LINK -DEIGEN_HAS_STD_RESULT_OF=0 -DEIGEN_MPL2_ONLY -DMYPROJECTCORE_DLL -DMYPROJECT_ENABLE_ASSERTIONS -DMYPROJECT_LOG_CAPTURE_ORIGIN -DNOMINMAX -DWIN32 -D_CRT_SECURE_NO_WARNINGS -D_MBCS -D_SCL_SECURE_NO_WARNINGS -D_SILENCE_CXX17_ADAPTOR_TYPEDEFS_DEPRECATION_WARNING -D_SILENCE_CXX17_CODECVT_HEADER_DEPRECATION_WARNING -D_USE_MATH_DEFINES -D_WINDLL -I/home/marco/projects/myproject/myprojectwithplugins/cmake-build-msvc-release/MyProjectSuite/MyProjectLib/MyProjectCore/MyProjectCore_autogen/include -I/home/marco/projects/myproject/myprojectwithplugins/MyProjectSuite/Core/include -I/home/marco/projects/myproject/myprojectwithplugins/MyProjectSuite/Core/ext -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/eigen/3.4.0 -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/zlib/1.2.11/include -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/boost/1.70.0 -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/zstd/1.5.0/include -external:I/mnt/windows/Dev/myproject/externals/externals-vs2019-x64/minizip/3.0.9/include -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/pcg-cpp/0.98.1.1-ffd522e7 -external:I/mnt/windows/Dev/myproject/externals/externals-vs2017-x64/date-tz/3.0.1/include -external:W0 /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MD /O2 /Ob2 /DNDEBUG -std:c++17 /W4 /WX /experimental:external /external:W3 /Oi /bigobj /EHsc /FC /MP /wd4068 /wd4251 /wd4800 /wd4324 /Z7 /diagnostics:caret -openmp

huangqinjin · 2023-12-16T01:37:14Z

@marcoesposito1988 thanks. Could you please also provide an example of link.exe?

marcoesposito1988 · 2023-12-16T01:52:35Z

@marcoesposito1988 thanks. Could you please also provide an example of link.exe?

Here:

MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/MyProjectCore_autogen/mocs_compilation.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Assert.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/ByteBuffer.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/ByteBufferStream.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/ByteSize.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Compression.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Configurable.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Cryptography.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/DateTime.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/DynamicLib.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Encoding.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Hashing.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Log.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Parameter.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Platform.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Properties.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/PropertiesIO.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Random.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/RandomEngine.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Signal.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/String.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Threading.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Timer.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Zip.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Log/ConsoleSink.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Log/FileSink.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Log/Sink.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Log/Worker.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/Directory.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/File.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/Filesystem.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/Path.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/TemporaryFile.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/TemporaryDirectory.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Filesystem/Url.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Circle.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Cuboid.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Ellipse.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Intersection.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Line.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/LineHelpersImpl.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Plane.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Polygon.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Rectangle.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Geometry/Triangle.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Resource/FilesystemRepository.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Resource/Resource.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Threading/PlaybackTimer.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Threading/StoppableThread.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Threading/ThreadPool.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/src/Utils/Version.cpp.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/ext/monocypher/monocypher.c.obj
MyProjectSuite/MyProjectLib/MyProjectCore/CMakeFiles/MyProjectCore.dir/MyProjectVersion.rc.res  Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/zlib/1.2.11/lib/zlib.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_thread-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_filesystem-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_date_time-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_system-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_chrono-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_regex-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_locale-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_program_options-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/zstd/1.5.0/lib/zstd_static.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2019-x64/minizip/3.0.9/lib/libminizip.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2019-x64/minizip/3.0.9/lib/bzip2.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2019-x64/minizip/3.0.9/lib/liblzma.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/date-tz/3.0.1/lib/Release/date-tz.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/boost/1.70.0/lib/libboost_atomic-vc141-mt-x64-1_70.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/zlib/1.2.11/lib/zlib.lib   Z:/mnt/windows/Dev/myProject/externals/externals-vs2017-x64/zstd/1.5.0/lib/zstd_static.lib   kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib

huangqinjin · 2023-12-16T02:23:20Z

@marcoesposito1988 Thanks again.

I think the regex can be simplified to s@(^|[[:blank:]])((/[^/[:blank:]]*){2,})@\1z:\2@g
We care about BSD sed (e.g. macOS). So referring to https://stackoverflow.com/a/24276470, we should

Use -E instead of -r.

Handle -i flag like

msvc-wine/wrappers/cl

Lines 42 to 47 in 539b98e

    
           if sed --help 2>&1 | grep '\-i extension' >/dev/null; then 
        
               inplace=(-i '') # BSD sed 
        
           else 
        
               inplace=(-i)    # GNU sed 
        
           fi 
        
           sed "${inplace[@]}" -E 's/\r//;'"$unixify_line" "$arg_Fi"

Since link.exe options are in form of /OPTION:arg, and colon is illegal for Windows paths, colon can be added to [^/:[:blank:]]. Such that we don't wrongly convert /LIBPATH:/mnt to z:/LIBPATH:/mnt. @mstorsjo @marcoesposito1988 your opinion?

mstorsjo · 2023-12-21T10:31:51Z

@marcoesposito1988 Thanks again.

I think the regex can be simplified to s@(^|[[:blank:]])((/[^/[:blank:]]*){2,})@\1z:\2@g

Hmm, why do you want the {2,} part here - why do we need at least two path segments before doing this rewrite? Thinking out loud, I guess that you want it so that you don't accidentally rewrite an option like /MD into z:/MD? But what about e.g. /Fo/absolute/path?

Since link.exe options are in form of /OPTION:arg, and colon is illegal for Windows paths, colon can be added to [^/:[:blank:]]. Such that we don't wrongly convert /LIBPATH:/mnt to z:/LIBPATH:/mnt. @mstorsjo @marcoesposito1988 your opinion?

That's probably a good thing to do.

FWIW, in the existing rewrite for actual arguments, we have a bit of a heuristic for what to convert; if $(dirname $potentialPath) is an existing directory (and not /) we do convert it. Here it's probably not quite as nice to do such heuristics.

I'd like to see it written out, either here in the PR or perhaps rather as a code comment, what cases the regex is designed around and how it tries to handle them.

huangqinjin · 2023-12-21T11:03:19Z

But what about e.g. /Fo/absolute/path?

To keep it simple and correct, I think we should restrict the rewritten to only link.exe (probably lib.exe as well). Because:

The original issue is about link.exe.
cl.exe accepts /Fo/absolute/path.

mstorsjo · 2023-12-21T11:05:12Z

But what about e.g. /Fo/absolute/path?

To keep it simple and correct, I think we should restrict the rewritten to only link.exe (probably lib.exe as well). Because:

The original issue is about link.exe.

cl.exe accepts /Fo/absolute/path.

That sounds reasonable - there's much more seldom any need for response files for compilation. It'd be good to clarify these design aspects in a code comment.

huangqinjin · 2023-12-21T11:17:20Z

Then we need a way to detect which tool is running in wine-msvc.sh.

mstorsjo · 2023-12-21T11:37:23Z

Then we need a way to detect which tool is running in wine-msvc.sh.

Ah, right. Or what if we just have patterns that primarily match linker commands, is there any risk of it breaking anything if it would be applied on cl.exe? Or is it just the case that we wouldn't end up handling all the tricky cases in cl.exe command lines?

huangqinjin · 2023-12-21T12:16:25Z

Then we need a way to detect which tool is running in wine-msvc.sh.

Ah, right. Or what if we just have patterns that primarily match linker commands, is there any risk of it breaking anything if it would be applied on cl.exe? Or is it just the case that we wouldn't end up handling all the tricky cases in cl.exe command lines?

We are rewriting /absolute/path in link.exe command line, which is hard to distinguish from /Fo/absolute/path in cl.exe command line. So using tool name is much simple.

mstorsjo · 2023-12-21T12:17:39Z

Then we need a way to detect which tool is running in wine-msvc.sh.

Ah, right. Or what if we just have patterns that primarily match linker commands, is there any risk of it breaking anything if it would be applied on cl.exe? Or is it just the case that we wouldn't end up handling all the tricky cases in cl.exe command lines?

We are rewriting /absolute/path in link.exe command line, which is hard to distinguish from /Fo/absolute/path in cl.exe command line. So using tool name is much simple.

Right, that's true - I agree.

marcoesposito1988 · 2025-04-15T11:26:27Z

Hi @mstorsjo and @huangqinjin,

I updated the PR with the suggestions I got from our discussion (using portable sed flags and the new regex, only apply the change when the tool is link.exe or lib.exe).

Can you please review the changes again?

mstorsjo mentioned this pull request Feb 1, 2024

Wine: "build_command_line command line too long" #105

Open

huangqinjin mentioned this pull request Jun 10, 2024

Support manifest filepaths #130

Merged

marcoesposito1988 and others added 3 commits April 15, 2025 13:16

Escape absolute paths with Z: also in response files

ef2391b

Replaced perl with sed, only process paths in *CMakeFiles*.rsp

b17dbf6

Use simplified regex and portable sed options

c6bcee4

marcoesposito1988 force-pushed the response_files branch from a4b37a1 to c6bcee4 Compare April 15, 2025 11:25

Uh oh!

Escape absolute paths with Z: also in response files #96

Are you sure you want to change the base?

Escape absolute paths with Z: also in response files #96

Uh oh!

Conversation

marcoesposito1988 commented Nov 27, 2023

Uh oh!

mstorsjo commented Dec 10, 2023

Uh oh!

huangqinjin commented Dec 14, 2023

Uh oh!

marcoesposito1988 commented Dec 15, 2023

Uh oh!

huangqinjin commented Dec 16, 2023

Uh oh!

marcoesposito1988 commented Dec 16, 2023

Uh oh!

huangqinjin commented Dec 16, 2023

Uh oh!

mstorsjo commented Dec 21, 2023

Uh oh!

huangqinjin commented Dec 21, 2023

Uh oh!

mstorsjo commented Dec 21, 2023

Uh oh!

huangqinjin commented Dec 21, 2023

Uh oh!

mstorsjo commented Dec 21, 2023

Uh oh!

huangqinjin commented Dec 21, 2023

Uh oh!

mstorsjo commented Dec 21, 2023

Uh oh!

marcoesposito1988 commented Apr 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants