RFR: race with nested repos in hgforest.sh (original) (raw)

David Holmes david.holmes at oracle.com
Sat Feb 2 00:37:45 UTC 2013

On 2/02/2013 9:06 AM, Chris Hegarty wrote:

On 1 Feb 2013, at 22:37, David Holmes<david.holmes at oracle.com> wrote:

Hi Chris,

On 2/02/2013 12:40 AM, Chris Hegarty wrote: [ to build-dev and core-libs-dev, expect reviewer from either, but will integrate through jdk8/tl ]

This issue is mainly of interest to Oracle engineers, but it effects the public hgforest script. When hgforest.sh is run with an addition argument to specify a closed server, there is a problem/race between the creation of the directories to hold nested repositories and the clone itself. These directories need to be created before the clone command is executed, otherwise it will fail, as below. I think I reported this myself just last week - probably internally to build or build-infra. Ah, I missed that. The weird thing is that based on the list of repos it shouldn't be grabbing the closed one yet. In my remote/slow environment I end up with all clones running concurrently.

I can see that happening if the sleeps are too short. But what I noticed from the logging was that the closed repos were being processed before the open ones - which shouldn't happen.

The trivial fix is to back off these nested repos until their containing directory exists. That seems to assume/require that the nested repo will always be later in the list. They happen to be but .... No so. It will just wait for the process to clone the outer repo to start.

Ok. Some clone will make progress and so we should ultimately get to start the outer repo.

Thanks, David

Workaround is to run getsource.sh twice first without the "extra base" then with it. Good suggestion. -Chris

David -----

Webrev: http://cr.openjdk.java.net/~chegar/hgforestnestedRepos/webrev/ sh ./getsource.sh http://xxx.yyy.oracle.com |& tee clone.log # Repositories: corba jaxp jaxws langtools jdk hotspot jdk/src/closed jdk/make/closed jdk/test/closed hotspot/make/closed hotspot/src/closed hotspot/test/closed deploy install sponsors pubs Waiting 5 secs before spawning next background command. Waiting 5 secs before spawning next background command. Waiting 5 secs before spawning next background command. jdk/src/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u /java/devtools/sparc/mercurial/latest/bin/hg clone http://xxx.yyy.oracle.com/jdk8/tl/jdk/src/closed jdk/src/closed jdk/src/closed: abort: No such file or directory: jdk/src/closed jdk/make/closed: /java/devtools/sparc/mercurial/0.9.5/bin/python -u /java/devtools/sparc/mercurial/latest/bin/hg clone http://xxx.yyy.oracle.com/jdk8/tl/jdk/make/closed jdk/make/closed jdk/make/closed: abort: No such file or directory: jdk/make/closed Waiting 5 secs before spawning next background command. ....

-Chris.

More information about the core-libs-dev mailing list