RFR: race with nested repos in hgforest.sh (original) (raw)

David Holmes david.holmes at oracle.com
Wed Feb 6 01:53:40 UTC 2013

Hi Chris,

I don't speak fluent sh but ...

204 # Terminate with exit 0 only if all subprocesses were successful 205 ec=0 206 if [ -d ${tmp} ]; then 207 for rc in ls ${tmp}/*.pid.rc ; do

I don't think you need to subshell an invocation of ls, simply do

for rc in "${tmp}/*.pid.rc" ; do

208 exit_code=cat ${rc} | tr -d ' \n\r' 209 if [ "${exit_code}" != "0" ] ; then 210 echo "WARNING: ${rc} exited abnormally."

Not sure printing ${rc} will be very informative there ??

211 ec=1

Is there a "break" we can add here?

In get_source.sh

Get clones of all nested repositories

! sh ./common/bin/hgforest.sh clone "$@" || exit 1

Update all existing repositories to the latest sources

sh ./common/bin/hgforest.sh pull -u

don't we want "|| exit 1" for the pull case as well?

Thanks, David

On 6/02/2013 12:07 AM, Chris Hegarty wrote:

On 02/05/2013 06:21 AM, David Holmes wrote:

Chris,

When these failures occur does the failure get reflected in an error exit code? Not currently, good point. I'm seeing hudson builds merilly buildinh OpenJDK instead of Oracle JDK because the closed repos were skipped due to this bug :( This is what I am seeing too, and the reason I proposed a patch to resolve it. Updated webrev; http://cr.openjdk.java.net/~chegar/hgforestnestedRepos/webrev/ 1) now checks, and returns, appropriate error code, so that higher level scripts can determine if it completed successfully 2) Added diagnostic error message to help identify potential future problems if a nested repo is blocked for too long -Chris.

David On 2/02/2013 12:40 AM, Chris Hegarty wrote: [ to build-dev and core-libs-dev, expect reviewer from either, but will integrate through jdk8/tl ]

This issue is mainly of interest to Oracle engineers, but it effects the public hgforest script. When hgforest.sh is run with an addition argument to specify a closed server, there is a problem/race between the creation of the directories to hold nested repositories and the clone itself. These directories need to be created before the clone command is executed, otherwise it will fail, as below. The trivial fix is to back off these nested repos until their containing directory exists. Webrev: http://cr.openjdk.java.net/~chegar/hgforestnestedRepos/webrev/ sh ./getsource.sh http://xxx.yyy.oracle.com | & tee clone.log # Repositories: corba jaxp jaxws langtools jdk hotspot jdk/src/closed jdk/make/closed jdk/test/closed hotspot/make/closed hotspot/src/closed hotspot/test/closed deploy install sponsors pubs Waiting 5 secs before spawning next background command. Waiting 5 secs before spawning next background command. Waiting 5 secs before spawning next background command. jdk/src/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u /java/devtools/sparc/mercurial/latest/bin/hg clone http://xxx.yyy.oracle.com/jdk8/tl/jdk/src/closed jdk/src/closed jdk/src/closed: abort: No such file or directory: jdk/src/closed jdk/make/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u /java/devtools/sparc/mercurial/latest/bin/hg clone http://xxx.yyy.oracle.com/jdk8/tl/jdk/make/closed jdk/make/closed jdk/make/closed: abort: No such file or directory: jdk/make/closed Waiting 5 secs before spawning next background command. ....

-Chris.

More information about the core-libs-dev mailing list