[PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows (original) (raw)

Andrew Luo andrewluotechnologies at outlook.com
Wed Dec 19 18:40:17 UTC 2018

Hi Erik,

Which target are you using (make exploded-image?)? I never saw this error while building on my machine (I've built about 10 times now, I'm on Windows 10 1809 for what it's worth). Perhaps I can try to reproduce this on my system as well...

Thanks,

-Andrew

-----Original Message----- From: Erik Joelsson <erik.joelsson at oracle.com> Sent: Wednesday, December 19, 2018 8:28 AM To: Andrew Luo <andrewluotechnologies at outlook.com>; Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> Cc: build-dev at openjdk.java.net Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows

I'm now seeing intermittent build failures that look like this:

make[3]: *** No rule to make target '/mnt/d/erik/jdk-sandbox/open/src/java.security.jgss/share/classes/sun/security/krb5/internal/TGSReq.java', needed by '/mnt/d/erik/jdk-sandbox/build/windows-x86_64-server-release/jdk/modules/java.security.jgss/_the.java.security.jgss_batch'. Stop.

The particular file that's missing varies, and deleting the output dir for that module and rebuild works. The common pattern seems to be upper case letters in the file name of the source file.

I will investigate some more.

/Erik

On 2018-12-19 06:18, Erik Joelsson wrote:

I can also report that on the Windows 10 machine I'm testing this on, "make bundles" on Cygwin took 12m56s and in WSL 8m39s, so a pretty big improvement!

/Erik On 2018-12-19 03:45, Erik Joelsson wrote: Hello,

On 2018-12-19 00:19, Erik Joelsson wrote: Hello Andrew,

On 2018-12-18 12:45, Andrew Luo wrote: Hi Erik/Magnus,

I've attached my latest changes: 1. Fixed a file I forgot to revert in my previous change while trying something out... 2. Added information about case sensitivity of the OpenJDK build directory (yes, I did use the make target to generate the HTML file) 3. Fixed Cygwin (hopefully, I don't have a Cygwin environment to verify this) I will take this patch for a spin and see. After applying a fix for https://bugs.openjdk.java.net/browse/JDK-8215635, I managed to build everything as well. I pushed some minor adjustments to make Cygwin work too. I will need to take this through more thorough testing. /Erik With this patch I've tested the following targets: exploded-image (default): Works images: Works bundles: Works test: Completes, but some tests fail. I didn't go through the test failures completely, but all of the ones I did see are due to attempting to call CreateProcess with "sh" as the argument (in jtreg: http://hg.openjdk.java.net/code-tools/jtreg/file/36c592d2f544/src/share/classes/com/sun/javatest/regtest/exec/ShellAction.java). This isn't supported in Windows, perhaps using a Linux boot JDK would fix this (but might break other things).  I can look into fixing it (on WSL you can call "wsl sh", for example), but I think since it's a completely separate repo anyways, it would be best to take up those changes separately.  Let me know your thoughts on this. Ah, if a Java process is launched from a Cygwin environment, it will have the unix/cygwin tools in the path so those can be launched directly. When running in WSL, it will launch the Windows binary java.exe in the Windows environment so there is no trace of WSL. I agree that we can look into this later, but we need to note that running tests is not completely supported in WSL. /Erik Otherwise, let me know if there is any other comments/suggestions before we can merge this into the main repository. Thanks, -Andrew -----Original Message----- From: Erik Joelsson <erik.joelsson at oracle.com> Sent: Monday, December 17, 2018 9:52 AM To: Andrew Luo <andrewluotechnologies at outlook.com>; Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> Cc: build-dev at openjdk.java.net Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows Hello Andrew, On 2018-12-16 00:01, Andrew Luo wrote: For me, /mnt/c was already mounted case insensitive.  Maybe this is only the default for the C:\ drive though (or perhaps depends on your Windows/WSL version?) I think the default is "dir", which will cause any new directory created from WSL to be case sensitive, so I would say this needs to be documented in building.md. Anyways, I've synced down the sandbox and added a new patch to address some of the feedback (and some of my own minor enhancements):

1. Got rid of EXECUTABLESUFFIX in favor of EXESUFFIX, which had to move earlier in the sequence 2. Use $EXESUFFIX instead of .exe literal per Magnus' feedback 3. Added information about WSL versioning to build, similar to Cygwin 4. Updated building.md and building.html with WSL build instructions Nice! I've applied and pushed this patch to the sandbox. Just to be sure, did you generate the html version with pandoc using "make update-build-docs"? If not, we will need to make sure that is done before the final push. I noticed trailing whitespace in some files. Jcheck will reject that in most types of files but in the build, we are a bit on our own trying to avoid it. (By the way, you misspelled my name in your sandbox commit): http://hg.openjdk.java.net/jdk/sandbox/rev/12615de8335e Terribly sorry about that! The combination of u and o is a common slip for me on the keyboard. It's correct in the new commit at least, and in the final commit, contributors are attributed with email addresses. I will work on fixing the Cygwin path extraction in my next patch.  Most likely I will just restore the old code for Cygwin while using the new code for WSL, unless there are other suggestions... Aside from this, is there any other feedback that I should take into account before we can merge this into the main repository? That may be the best solution. I tried to build some more targets and failed. Please make sure you can do "make bundles". That will build docs and tests in addition to just the product and also do the packaging into zip/tar.gz. It would also be nice if "make test" worked. Note that Magnus is now on vacation and I will be traveling, so you will not hear from me until Wednesday. /Erik Thanks, -Andrew -----Original Message----- From: Erik Joelsson <erik.joelsson at oracle.com> Sent: Friday, December 14, 2018 5:42 PM To: Andrew Luo <andrewluotechnologies at outlook.com>; Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> Cc: build-dev at openjdk.java.net Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows After having configured my WSL to mount using case=off, I was able to successfully build images using the latest patch as applied in the sandbox. /Erik On 2018-12-14 17:23, Erik Joelsson wrote: Hello again,

I took the liberty of creating a bug for this and also a sandbox branch where I've applied your latest patch. If you clone that you can send further patches based on a known state in the sandbox. This will make it easier to see what you are actually doing in each update, as well as give us better references when discussing them. It also gives me the ability to directly change things so we can keep Cygwin/msys working. https://bugs.openjdk.java.net/browse/JDK-8215445 http://hg.openjdk.java.net/jdk/sandbox/shortlog/12615de8335e /Erik On 2018-12-14 16:47, Erik Joelsson wrote: Hello,

You beat me to it. I just found the rc.exe problem was that FIXPATHPATH in spec.gmk.in was quoted. Make just propagates quotes verbatim, so then fixpath.c would create a path variable like; $PATH;"$FIXPATHPATH" Which is why link.exe could not find rc.exe. /Erik On 2018-12-14 16:32, Andrew Luo wrote: Ok, here's my latest patch (I didn't add your case sensitivity fix yet, but will do next patch).  I believe this should get you past the rc.exe issues.

Thanks, -Andrew -----Original Message----- From: Erik Joelsson <erik.joelsson at oracle.com> Sent: Friday, December 14, 2018 4:15 PM To: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> Cc: Andrew Luo <andrewluotechnologies at outlook.com>; build-dev at openjdk.java.net Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows

On 2018-12-14 16:06, Magnus Ihse Bursie wrote: 14 dec. 2018 kl. 23:42 skrev Erik Joelsson <erik.joelsson at oracle.com>: I found the reason it's not failing make. It returns "1" and NativeCompilation.gmk currently ignores 1 explicitly for Visual Studio. I added that back in 2014 in https://bugs.openjdk.java.net/browse/JDK-8065576, but I can't figure out why. Nothing mentioned in either comment or review. Sounds like it's ripe for removal then. :) I wonder what kind of issue you might have run into that caused a returned 1 to happen and yet we didn't want to consider it a failure... If I'm to guess, I think it's one of the commands we pipe the output to when the output is zero. This would explain why it was added together with pipefail. /Erik /Magnus /Erik On 2018-12-14 13:59, Magnus Ihse Bursie wrote:

On 2018-12-14 22:15, Erik Joelsson wrote: I get the same error for pch and it still continues, but this time I let it run until it eventually fails for real when it can't link. Perhaps it's simply cl.exe that isn't returning non zero for this error? When the linker fails, make fails, so propagation doesn't seem broken. That does also seem really weird, considering that it claims it to be a "fatal error". Can you repeat the command at the command line and get the failure again, and then check the return value? Can you rewrite the command line and run it from the devenv prompt? That is, is there any indication that the pch file itself is messed up, or can it be used if running the compilation that should use it from an "ok" prompt? /Magnus /Erik On 2018-12-14 12:55, Andrew Luo wrote: Hmm, I get the rc.exe error as well, but now it is much later down the line... Still investigating... Thanks, -Andrew -----Original Message----- From: Andrew Luo Sent: Friday, December 14, 2018 12:34 PM To: 'Andrew Luo' <andrewluotechnologies at outlook.com>; Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; Erik Joelsson <erik.joelsson at oracle.com> Cc: build-dev at openjdk.java.net Subject: RE: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows Try this updated patch with some fixes... Thanks, -Andrew -----Original Message----- From: build-dev <build-dev-bounces at openjdk.java.net> On Behalf Of Andrew Luo Sent: Friday, December 14, 2018 12:01 PM To: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; Erik Joelsson <erik.joelsson at oracle.com> Cc: build-dev at openjdk.java.net Subject: RE: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows I think I have a fix for it.  Give me a minute (or a few hours depending on if it works). Thanks, -Andrew -----Original Message----- From: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com> Sent: Friday, December 14, 2018 11:42 AM To: Erik Joelsson <erik.joelsson at oracle.com> Cc: Andrew Luo <andrewluotechnologies at outlook.com>; build-dev at openjdk.java.net Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows

14 dec. 2018 kl. 20:31 skrev Erik Joelsson <erik.joelsson at oracle.com>:

On 2018-12-14 11:05, Magnus Ihse Bursie wrote:

On 2018-12-14 19:41, Erik Joelsson wrote: On 2018-12-14 10:28, Magnus Ihse Bursie wrote:

On 2018-12-14 19:23, Erik Joelsson wrote: Hello, I took your patch for a spin, and configure passes, but I get the same build error I got with my patch: fatal error C1083: Cannot open compiler intermediate file: _'d:\erik\jdk-wsl\build\windows-x8664-server-release_ hotsp _ot_ varia nt-server\libjvm\objs\buildlibjvm.pch': No such file or directory This is repeated for every C++ file in Hotspot. I see two issues here. First of all, I need to figure out why the compiler will not find the file, which is clearly there. Second, why isn't this failure picked up by make? Somewhere the return value of cl.exe is disappearing. Can you build without errors if you disable PCH? Could you? That is, is it only the PCH that is problematic? Trying that now. Also, a wild guess: can it be related to file permissions? Can you read the file properly from both WSL and Windows? It is readable, but it could be something with case. The file is actually called BUILDLIBJVM.pch, but that is also how it's given to the compiler command line. Here is the output from DEBUGFIXPATH: Weird. What if you, after a failed build, rename it to buildlibjvm.pch? Doing that causes a new error: d:\erik\jdk-wsl\open\src\hotspot\share\gc\shared\accessBarrierSupport. cpp : fatal error C1382: the PCH file 'd:\erik\jdk-wsl\build\windows-x8664-server-release\hots pot\v ari ant-s erver\libjvm\objs\buildlibjvm.pch' has been rebuilt since 'd:\erik\jdk-wsl\build\windows-x8664-server-release\hots pot\v ari ant-s erver\libjvm\objs\accessBarrierSupport.obj' was generated. Please rebuild this object But I think even more important is that make is not getting the error. The build just continues until interrupted. Agree, that's bad. Does fixpathdebug print exit code? If so, what does it say? If not, we should add that instrumentation. /Magnus Compiling adx86expand.cpp (for jvm.dll) fixpath input line -wsl\build\windows-x8664-server-release\configure-sup _port_ bin \fixp ath.exe -w This starts out quite odd..? -wsl\build...? I agree, didn't look into that part. /mnt/c/PROGRA2/MICROS1/2017/PROFES1/VC/Tools/MSVC/14161. 270 /bin/ Hostx86/x64/cl.exe Also, FWIW, this seems not to have been properly case treated. Which version of the patch are you using? The last one posted by Andrew: "diff15.txt". /Erik /Magnus -showIncludes -Fp/mnt/d/erik/jdk-wsl/build/windows-x8664-server-rele ase/h ots pot/v ariant-server/libjvm/objs/BUILDLIBJVM.pch -Yuprecompiled.hpp -D_STDCFORMATMACROS -D_STDCLIMITMACROS -D_STDCCONSTANTMACROS -DNOMINMAX -DWIN32LEANANDMEAN -nologo -MD -MP -DWINDOWS -DWIN32 -DJNIIMPLEMENTATION -W3 -DVMLITTLEENDIAN -DLP64=1 -DPRODUCT -DTARGETARCHx86 -DINCLUDESUFFIXOS=windows -DINCLUDESUFFIXCPU=x86 -DINCLUDESUFFIXCOMPILER=visCPP -DTARGETCOMPILERvisCPP -DAMD64 "-DHOTSPOTLIBARCH="amd64"" -DCOMPILER1 -DCOMPILER2 -DINCLUDEZGC=0 -I/mnt/d/erik/jdk-wsl/build/windows-x8664-server-relea se/ho tsp ot/va riant-server/gensrc/adfiles -I/mnt/d/erik/jdk-wsl/closed/src/hotspot/share -I/mnt/d/erik/jdk-wsl/open/src/hotspot/share -I/mnt/d/erik/jdk-wsl/open/src/hotspot/os/windows -I/mnt/d/erik/jdk-wsl/open/src/hotspot/cpu/x86 -I/mnt/d/erik/jdk-wsl/open/src/hotspot/oscpu/windowsx 86 -I/mnt/d/erik/jdk-wsl/build/windows-x8664-server-relea se/ho tsp ot/va riant-server/gensrc -I/mnt/d/erik/jdk-wsl/open/src/hotspot/share/precompile d -I/mnt/d/erik/jdk-wsl/open/src/hotspot/share/include -I/mnt/d/erik/jdk-wsl/open/src/hotspot/os/windows/inclu de -I/mnt/d/erik/jdk-wsl/build/windows-x8664-server-relea se/su ppo rt/mo dulesinclude/java.base -I/mnt/d/erik/jdk-wsl/build/windows-x8664-server-relea se/su ppo rt/mo dulesinclude/java.base/win32 -I/mnt/d/erik/jdk-wsl/open/src/java.base/share/native/l ibjim age -Z7 -d2Zi+ -wd4800 -WX -I/mnt/c/PROGRA2/MICROS1/2017/PROFES~1/VC/Tools/MSVC/ 1416~ 1.2 70/at lmfc/include -I/mnt/c/PROGRA2/MICROS1/2017/PROFES~1/VC/Tools/MSVC/ 1416~ 1.2 70/in clude -I/mnt/c/PROGRA2/WI3CF21/10/Include/100177~1.0/ucrt -I/mnt/c/PROGRA2/WI3CF21/10/Include/100177~1.0/shared -I/mnt/c/PROGRA2/WI3CF21/10/Include/100177~1.0/um -I/mnt/c/PROGRA2/WI3CF21/10/Include/100177~1.0/winrt -I/mnt/c/PROGRA2/WI3CF21/10/Include/100177~1.0/cppwin rt -O2 -Oy- "-DTHISFILE=""" -c -Fo/mnt/d/erik/jdk-wsl/build/windows-x8664-server-rele ase/h ots pot/v ariant-server/libjvm/objs/adx86expand.obj /mnt/d/erik/jdk-wsl/build/windows-x8664-server-release /hots pot /vari ant-server/gensrc/adfiles/adx86expand.cpp< fixpath using wsl mode, with path list: fixpath converted line c:/PROGRA2/MICROS1/2017/PROFES1/VC/Tools/MSVC/1416 1.270 /bi n/Hos tx86/x64/cl.exe -showIncludes -Fpd:/erik/jdk-wsl/build/windows-x8664-server-release/ hotsp ot/ varia nt-server/libjvm/objs/BUILDLIBJVM.pch -Yuprecompiled.hpp -D_STDCFORMATMACROS -D_STDCLIMITMACROS -D_STDCCONSTANTMACROS -DNOMINMAX -DWIN32LEANANDMEAN -nologo -MD -MP -DWINDOWS -DWIN32 -DJNIIMPLEMENTATION -W3 -DVMLITTLEENDIAN -DLP64=1 -DPRODUCT -DTARGETARCHx86 -DINCLUDESUFFIXOS=windows -DINCLUDESUFFIXCPU=x86 -DINCLUDESUFFIXCOMPILER=visCPP -DTARGETCOMPILERvisCPP -DAMD64 "-DHOTSPOTLIBARCH="amd64"" -DCOMPILER1 -DCOMPILER2 -DINCLUDEZGC=0 -Id:/erik/jdk-wsl/build/windows-x8664-server-release/h otspo t/v arian t-server/gensrc/adfiles -Id:/erik/jdk-wsl/closed/src/hotspot/share -Id:/erik/jdk-wsl/open/src/hotspot/share -Id:/erik/jdk-wsl/open/src/hotspot/os/windows -Id:/erik/jdk-wsl/open/src/hotspot/cpu/x86 -Id:/erik/jdk-wsl/open/src/hotspot/oscpu/windowsx86 -Id:/erik/jdk-wsl/build/windows-x8664-server-release/h otspo t/v arian t-server/gensrc -Id:/erik/jdk-wsl/open/src/hotspot/share/precompiled -Id:/erik/jdk-wsl/open/src/hotspot/share/include -Id:/erik/jdk-wsl/open/src/hotspot/os/windows/include -Id:/erik/jdk-wsl/build/windows-x8664-server-release/s uppor t/m odule sinclude/java.base -Id:/erik/jdk-wsl/build/windows-x8664-server-release/s uppor t/m odule sinclude/java.base/win32 -Id:/erik/jdk-wsl/open/src/java.base/share/native/libji mage -Z7 -d2Zi+ -wd4800 -WX -Ic:/PROGRA2/MICROS1/2017/PROFES~1/VC/Tools/MSVC/1416 ~1.27 0/a tlmfc /include -Ic:/PROGRA2/MICROS1/2017/PROFES~1/VC/Tools/MSVC/1416 ~1.27 0/i nclud e -Ic:/PROGRA2/WI3CF21/10/Include/100177~1.0/ucrt -Ic:/PROGRA2/WI3CF21/10/Include/100177~1.0/shared -Ic:/PROGRA2/WI3CF21/10/Include/100177~1.0/um -Ic:/PROGRA2/WI3CF21/10/Include/100177~1.0/winrt -Ic:/PROGRA2/WI3CF21/10/Include/100177~1.0/cppwinrt -O2 -Oy- "-DTHISFILE=""" -c -Fod:/erik/jdk-wsl/build/windows-x8664-server-release/ hotsp ot/ varia nt-server/libjvm/objs/adx86expand.obj d:/erik/jdk-wsl/build/windows-x8664-server-release/hot spot/ var iant- server/gensrc/adfiles/adx86expand.cpp< An interesting note is that make is rebuilding the pch file on every invocation so it too has trouble finding the file. /Erik

More information about the build-dev mailing list