Building OpenJDK9 on MSYS2 (original) (raw)

Peter Budai peterbudai at hotmail.com
Thu Oct 5 10:00:55 UTC 2017

Previous message (by thread): Building OpenJDK9 on MSYS2
Next message (by thread): Building OpenJDK9 on MSYS2
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Now the attachment as well…

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Peter Budai<mailto:peterbudai at hotmail.com> Sent: Thursday, October 5, 2017 11:59 AM To: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com>; Erik Joelsson<mailto:erik.joelsson at oracle.com>; build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net> Subject: RE: Building OpenJDK9 on MSYS2

Hi Magnus and Erik,

I really appreciate your quick feedback. I assumed that it won’t be easy, but I just don’t feel I should give up now - maybe later when I see the real scale of work. So bear with me for a time being.

Attached is a patch which already includes Magnus’ changes, plus a few which I have added:

basically enabling gcc for windows,
and modifying a logic for compiling fixpath (before that it was using hard-coded MS VSC compile flags)

So here is what I have as the result of configure:

The existing configuration has been successfully updated in /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release using configure arguments '--disable-freetype-bundling --disable-javac-server'.

Configuration summary:

Debug level: release
HS debug level: product
JDK variant: normal
JVM variants: server
OpenJDK target: OS: windows, CPU architecture: x86, address length: 64
Version string: 9-internal+0-adhoc.peterbud.jdk9 (9-internal)

Tools summary:

Environment: msys version 2.9.0(0.318/5/3) (root at /C/msys64)
Boot JDK: java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) (at /c/progra~~1/java/jdk18~~1.0_1)
Toolchain: gcc (GNU Compiler Collection)
C Compiler: Version 7.2.0 (at /C/msys64/mingw64/bin/gcc)
C++ Compiler: Version 7.2.0 (at /c/msys64/mingw64/bin/g++)

Build performance summary:

Cores to use: 4
Memory limit: 16216 MB

Its clear says that the toolchain is gcc 7.2 (BTW there is no Visual Studio on this machine)

Now for the details of the config log, you can see here: https://pastebin.com/MN2ZYcHH

And about the build process and the error I get:

$ make JOBS=1 Building target 'default (exploded-image)' in configuration 'windows-x86_64-normal-server-release' Compiling 8 files for BUILD_TOOLS_LANGTOOLS Compiling 17 properties into resource bundles for jdk.compiler Parsing 1 properties into enum-like class for jdk.compiler Compiling 19 properties into resource bundles for jdk.javadoc Compiling 12 properties into resource bundles for jdk.jdeps Compiling 7 properties into resource bundles for jdk.jshell Compiling 117 files for BUILD_INTERIM_java.compiler Compiling 396 files for BUILD_INTERIM_jdk.compiler Compiling 61 files for BUILD_INTERIM_jdk.jdeps Compiling 457 files for BUILD_INTERIM_jdk.javadoc Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Compiling 159 files for BUILD_TOOLS_JDK Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java] Error 1 make[3]: *** Deleting file '/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java' make[2]: *** [make/Main.gmk:115: java.base-gensrc-jdk] Error 2

ERROR: Build failed for target 'default (exploded-image)' in configuration 'windows-x86_64-normal-server-release' (exit code 2)

No indication of failed target found. Hint: Try searching the build log for '] Error'. Hint: See common/doc/building.html#troubleshooting for assistance.

make[1]: *** [/home/peterbud/jdk9/make/Init.gmk:296: main] Error 2 make: *** [/home/peterbud/jdk9/make/Init.gmk:185: default] Error 2

If I run here make JOBS=1 LOG=debug The failing line seems to be this:

( /usr/bin/gawk '/@@END_COPYRIGHT@@/{exit}1' /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template && /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe -m/C/msys64/@/c/msys64/@/c/progra~ /C/msys64/mingw64/bin/gcc -E -x c /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template 2> >(/usr/bin/grep -v '^SocketOptionRegistry.java.template$' >&2) | /usr/bin/gawk '/@@START_HERE@@/,0' | /usr/bin/sed -e 's/@@START_HERE@@/// AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' -e 's/PREFIX_//' -e 's/^#.*//' ) > /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java] Error 1

Now the interesting is: if I copy this line above to the bash prompt, it runs without problem, and the file support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java Is produced.

Then I can again issue make JOBS=1 LOG=debug

And the compile process is being continued, until a similar error pops up with a different generated file. I have an assumption that this happens because make is still running parallel jobs, despite JOBS=1 but I’m not sure.

How could I best tackle this?

Thank you and best regards,

Peter

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

From: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com> Sent: Thursday, October 5, 2017 11:33 AM To: Erik Joelsson<mailto:erik.joelsson at oracle.com>; Peter Budai<mailto:peterbudai at hotmail.com>; build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net> Subject: Re: Building OpenJDK9 on MSYS2

On 2017-10-05 10:10, Erik Joelsson wrote:

Hello Peter,

On 2017-10-04 21:15, Peter Budai wrote: Hi Magnus,

Thanks for the quick reply I’ll check these patches with msys2. Let me specify with more details what I’d like to achieve: I’d like to build OpenJDK9 with MSYS2 MINGW64 environment using gcc toolchain. (I’m not sure how familiar are you with MSYS2, but there are 3 different environments: MSYS2, MINGW32 and MINGW64). In theory MINGW64 with gcc is the closes you can get on Windows platform as a gcc unix like build environment, which produces still a native 64-bit executable on Windows. I’m not very familiar with OpenJDK yet, so therefore I’d like to hear your opinion: how realistic is that? Sorry to disappoint, but I would say that requires major work. There is a strong historic assumption that windows builds are done using Visual Studio. We have abstracted away some of it in configure (see TOOLCHAINTYPE), but it's very far from enough to change compiler environment for a Windows build. The native sources are also bound to make a lot of such assumptions. I would expect the changes needed to be in the thousands of lines of code.

I agree that it requires hard work (even if "thousands" might be an overestimation I think, but "hundreds" is not enough, so it's the right magnitude). On the other hand, it would be really good if we did sort things out, so that we had proper conditions based on OS vs compiler/toolchain.

If you really want to start, the first thing is to patch toolchain.m4 to VALID_TOOLCHAINS_windows="microsoft gcc" and then call configure using "bash configure --with-toolchain-type=gcc".

As Erik, I doubt you will come very far before things starts tumbling down.

When we say supporting the build in msys2 instead of cygwin, we just mean using msys2 as the unix emulating layer for our tools like make/bash/grep/sed etc. One think I have done successfully is running the build in WSL (Windows Subsystem for Linux), but that isn't all that helpful as WSL for practical purposes is more or less like running Linux in a VM, so the build sees a Linux system and builds a Linux binary. As a side note: with MINGW64 I have managed to run configure phase successfully for OpenJDK. The compile process has also started and went for a while, but interestingly I run into some kind of race conditions as make stopped with an error. Using LOG=debug I have fond the failing line and then copying the failed command and pasting it to the bash prompt it successfully generated the output target, and then the build process run further when a similar situation happened. Also pasting the failed command run in the bash without any problem, and build continued… until the next. Without seeing the errors I can't say much. I very much doubt that you are running with gcc as the compiler though. Configure isn't easily fooled into using a different compiler to what it prefers, and I would expect things to crash and burn pretty early if you actually did. /Erik

I have tried to run make JOBS=1, but did not help, strangely I have still seen in the log make[3] and make[4] logs which suggested that there are more than one make jobs were running. Also tried .configure --with-output-sync=recurse without success (same symptoms) Let me know your thoughts. Best regards, Peter Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com> Sent: Wednesday, October 4, 2017 1:04 AM To: Peter Budai<mailto:peterbudai at hotmail.com>; build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net> Subject: Re: Building OpenJDK9 on MSYS2 Actually, it wasn't so much remaining trouble. :-) I fired up msys2 and checked out where I left off. It turned out that the remaining snag was that msys2 tries to convert command lines automatically, from "unix" style paths to "windows" style paths. Unfortunately, it does not do this very well and it breaks all sorts of things. We already have a FIXPATH solution in place which deals with this, so basically all I had to do was disable this (by setting MSYS2ARGCONVEXCL to "*"). However, this broke our cygpath replacement hack (!) so I had to disable it there. Sigh. Anyway, with those fixes it ran and worked well. (I also discovered and fixed a bug related to how we set up the FIXPATH variable on msys, but it only triggers in certain circumstances). With this patch I now jdk9 seems to build fine on msys2. It should apply cleanly on jdk9/jdk9. Since it turned out to be so trivial, I'll try to get it in in jdk10. Here's the patch if you want to apply it yourself: diff -r a08cbfc0e4ec common/autoconf/basicswindows.m4 --- a/common/autoconf/basicswindows.m4 Thu Aug 03 18:56:56 2017 +0000 +++ b/common/autoconf/basicswindows.m4 Wed Oct 04 00:53:58 2017 +0200 @@ -42,7 +42,7 @@ windowspath=$CYGPATH -m "$unixpath" $1="$windowspath" elif test "x$OPENJDKBUILDOSENV" = "xwindows.msys"; then - windowspath=cmd //c echo $unixpath + windowspath=MSYS2ARGCONVEXCL= cmd //c echo $unixpath $1="$windowspath" fi ]) @@ -136,6 +136,16 @@ fi ]) +ACDEFUN([BASICMSYSUPDATEFIXPATH], +[_ _+ # Take all collected prefixes and turn them into a_ _-m/c/foo@/c/bar at ... command line + # @ was chosen as separator to minimize risk of other tools messing around with it + alluniqueprefixes=echo "${allfixpathprefixes@<:@@@:>@}" \_ _+ | tr ' ' '\n' | <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>R</mi><mi>E</mi><msup><mi>P</mi><mrow><mo mathvariant="normal">′</mo><mi mathvariant="normal">/</mi></mrow></msup><mi mathvariant="normal">.</mi><msup><mi mathvariant="normal">/</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">GREP '^/./' | </annotation></semantics></math>GREP′/./′∣SORT | $UNIQ + fixpathargumentlist=echo $alluniqueprefixes | tr ' ' '@' + FIXPATH="$FIXPATHBIN -m$fixpathargumentlist" +]) + ACDEFUN([BASICFIXUPPATHMSYS], [ path="[$]$1" @@ -143,7 +153,7 @@ newpath="$path" if test "x$hascolon" = x; then # Not in mixed or Windows style, start by that. - newpath=cmd //c echo $path + newpath=MSYS2ARGCONVEXCL= cmd //c echo $path fi BASICMAKEWINDOWSSPACESAFEMSYS([$newpath]) @@ -155,6 +165,8 @@ # Save the first 10 bytes of this path to the storage, so fixpath can work. allfixpathprefixes=("${allfixpathprefixes@<:@@@:>@}" "${newpath:0:10}") + # We might need to re-evaluate FIXPATH. + BASICMSYSUPDATEFIXPATH ]) ACDEFUN([BASICFIXUPEXECUTABLECYGWIN], @@ -293,7 +305,7 @@ # Do not save /bin paths to allfixpathprefixes! else # Not in mixed or Windows style, start by that. - newpath=cmd //c echo $newpath + newpath=MSYS2ARGCONVEXCL= cmd //c echo $newpath BASICMAKEWINDOWSSPACESAFEMSYS([$newpath]) # Output is in $newpath BASICWINDOWSREWRITEASUNIXPATH(newpath) @@ -302,6 +314,8 @@ # Save the first 10 bytes of this path to the storage, so fixpath can work. allfixpathprefixes=("${allfixpathprefixes@<:@@@:>@}" "${newpath:0:10}") + # We might need to re-evaluate FIXPATH. + BASICMSYSUPDATEFIXPATH fi ]) @@ -347,6 +361,10 @@ WINDOWSENVVENDOR='msys' WINDOWSENVVERSION="$MSYSVERSION" + # Prohibit msys2 path conversion from trying to be "intelligent", and rely + # on fixpath instead. + export MSYS2ARGCONVEXCL="*" + ACMSGCHECKING([msys root directory as unix-style path]) # The cmd output ends with Windows line endings (CR/LF), the grep command will strip that away MSYSROOTPATH=cd / ; cmd /c cd | $GREP ".*" @@ -391,10 +409,7 @@ elif test "x$OPENJDKBUILDOSENV" = xwindows.msys; then # Take all collected prefixes and turn them into a -m/c/foo@/c/bar at ... command line # @ was chosen as separator to minimize risk of other tools messing around with it - alluniqueprefixes=echo "${allfixpathprefixes@<:@@@:>@}" \_ _- | tr ' ' '\n' | <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>R</mi><mi>E</mi><msup><mi>P</mi><mrow><mo mathvariant="normal">′</mo><mi mathvariant="normal">/</mi></mrow></msup><mi mathvariant="normal">.</mi><msup><mi mathvariant="normal">/</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">GREP '^/./' | </annotation></semantics></math>GREP′/./′∣SORT | $UNIQ - fixpathargumentlist=echo $alluniqueprefixes | tr ' ' '@' - FIXPATH="$FIXPATHBIN -m$fixpathargumentlist" + BASICMSYSUPDATEFIXPATH fi FIXPATHSRCW="$FIXPATHSRC" FIXPATHBINW="$FIXPATHBIN" diff -r a08cbfc0e4ec common/autoconf/build-aux/config.sub --- a/common/autoconf/build-aux/config.sub Thu Aug 03 18:56:56 2017 +0000 +++ b/common/autoconf/build-aux/config.sub Wed Oct 04 00:53:58 2017 +0200 @@ -30,7 +30,7 @@ DIR=dirname $0 # First, filter out everything that doesn't begin with "aarch64-" -if ! echo $* | grep '^aarch64-' >/dev/null ; then +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then . DIR/autoconf−config.sub"DIR/autoconf-config.sub "DIR/autoconf−config.sub"@" # autoconf-config.sub exits, so we never reach here, but just in # case we do: @@ -38,13 +38,17 @@ fi while test $# -gt 0 ; do - case $1 in + case $1 in -- ) # Stop option processing shift; break ;; aarch64-* ) config=echo $1 | sed 's/^aarch64-/arm-/' subargs="$subargs $config" shift; ;; + *-msys ) + config=echo $1 | sed 's/msys/mingw32/' + subargs="$subargs $config" + shift; ;; - ) # Use stdin as input. subargs="$subargs $1" shift; break ;; diff -r a08cbfc0e4ec common/autoconf/spec.gmk.in --- a/common/autoconf/spec.gmk.in Thu Aug 03 18:56:56 2017 +0000 +++ b/common/autoconf/spec.gmk.in Wed Oct 04 00:53:58 2017 +0200 @@ -120,6 +120,13 @@ # On Windows, the Visual Studio toolchain needs the PATH to be adjusted # to include Visual Studio tools (this needs to be in cygwin/msys style). export PATH:=@VSPATH@ + +endif + +ifeq ($(OPENJDKTARGETOSENV), windows.msys) + # On msys2, prohibit msys path conversion from trying to be + # "intelligent", and rely on fixpath instead. + export MSYS2ARGCONVEXCL:=* endif SYSROOTCFLAGS := @SYSROOTCFLAGS@ /Magnus On 2017-10-03 22:34, Magnus Ihse Bursie wrote: I gave msys2 a shot some time ago, but it ended up too much trouble. I'll share some of my notes from that attempt, for what it's worth.

To install package X/Y, run "pacman -S X/Y". Missing tools and packages where to find them: cmp: msys/diffutils tar: msys/tar make: msys/make unzip: msys/unzip zip: msys/zip config.sub reports msys as "x8664-pc-mingw32" but msys2 as "x8664-pc-msys". This patch adds postprocessing in "our" config.sub to report msys2 similar to msys. (Opinions, including my own :-) may vary if this really is the best way..) diff -r b88023f46daa common/autoconf/build-aux/config.sub --- a/common/autoconf/build-aux/config.sub Fri Jan 27 10:15:41 2017 +0100 +++ b/common/autoconf/build-aux/config.sub Fri Feb 03 05:00:25 2017 -0700 @@ -30,7 +30,7 @@ DIR=dirname $0 # First, filter out everything that doesn't begin with "aarch64-" -if ! echo $* | grep '^aarch64-' >/dev/null ; then +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then . DIR/autoconf−config.sub"DIR/autoconf-config.sub "DIR/autoconf−config.sub"@" # autoconf-config.sub exits, so we never reach here, but just in # case we do: @@ -45,6 +45,10 @@ config=echo $1 | sed 's/^aarch64-/arm-/' subargs="$subargs $config" shift; ;; + *-msys ) + config=echo $1 | sed 's/msys/mingw32/' + subargs="$subargs $config" + shift; ;; - ) # Use stdin as input. subargs="$subargs $1" shift; break ;; If I remember correctly, this got me past the configure stage at the time. I don't think it's very hard to get it to work on msys2, I just ran into one snag too many and didn't think msys2 would be used by anyone. /Magnus On 2017-10-03 17:20, Peter Budai wrote: Hello,

According to http://hg.openjdk.java.net/jdk9/jdk9/file/a08cbfc0e4ec/common/doc/building.html “msys2 and the new Windows Subsystem for Linux (WSL) would likely be possible to support in a future version but that would require a community effort to implement” I’d like to help making the OpenJDK 9 build working on msys2. What is the best way to move forward? Is there a similar effort in progress? Thank you and best regards, Peter

Previous message (by thread): Building OpenJDK9 on MSYS2
Next message (by thread): Building OpenJDK9 on MSYS2
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the build-dev mailing list