Possible open file leak in com.sun.tools.javac.file.JavacFileManager (original) (raw)

Jonathan Gibbons jonathan.gibbons at oracle.com
Tue Nov 13 21🔞44 UTC 2018

Andreas,

I investigated this using your test code, changing the location to use a search path given on the command line.

I see the effect are describing when the search path contains JAR files. There does not seem to be an issue when listing directories or system files.

You are right in that there is a missing 'close()', but I don't think it is in the list() method, as you suggested.  You don't give any significant amount of detail about the environment in which you are seeing this, although you do say "heavy use of compiling classes during runtime". That suggests to me that you are using the javax.tools API to run the compiler (as compared to command-line javac), which suggests that you may be setting up a file manager, as in your code example. If that is the case, a likely cause of your problems is that you are not closing any file managers created and used in your application.  When I modified my version of your code sample to close the file manager and run countOpenFiles() again, the number of open files reverted to the number at the beginning of the test.

You also note the change in behavior between JDK 8 and JDK 11. In JDK 9, javac was changed to use the new NIO-based APIs (java.nio.file.) instead of the old File-based APIs (java.io.). As part of this work, JAR files are now opened as NIO filesystems, and for performance reasons, they are held open until the file manager itself is closed. (Previously, javac had a custom implementation for handling ZIP files which did not rely on holding the JAR file open, meaning that you could get away with not bothering to close the file manager.)

So, you might try checking your code to ensure you are either reusing file managers, or else closing them when they are no longer required.

-- Jon

On 11/05/2018 06:24 AM, Andreas Fey wrote:

Hi all,

we maybe found a bug in the com.sun.tools.javac.file.JavacFileManager; our tool makes heavy use of compiling classes during runtime, and after switching from JDK 8 to 11, we noticed hundreds of open files being created during list() and only closed when JVM exists. The concerning lines of code are:  @Override @DefinedBy(Api.COMPILER)  public Iterable list(Location location,  String packageName,  Set<JavaFileObject.Kind> kinds,  boolean recurse)  throws IOException  {  checkNotModuleOrientedLocation(location);  // validatePackageName(packageName);  nullCheck(packageName);  nullCheck(kinds);  Iterable<? extends Path> path = getLocationAsPaths(location);  if (path == null)  return List.nil();  RelativeDirectory subdirectory = RelativeDirectory.forPackage(packageName);  ListBuffer results = new ListBuffer<>();  for (Path directory : path) {  Container container = getContainer(directory);  container.list(directory, subdirectory, kinds, recurse, results);  }  return results.toList();  } We think, a container.close() is missing in the for loop. Without this, a filehandle is created for every container/directiry found here. To test this, the following code snipped can be used, but the location must be set property to find a valid path != null: @Test public void testFileManager( ) throws IOException { countOpenFiles( ); final Set<javax.tools.JavaFileObject.Kind> kinds = new HashSet<>( ); kinds.add( javax.tools.JavaFileObject.Kind.OTHER ); kinds.add( javax.tools.JavaFileObject.Kind.SOURCE ); kinds.add( javax.tools.JavaFileObject.Kind.CLASS ); kinds.add( javax.tools.JavaFileObject.Kind.HTML ); final StandardJavaFileManager sfm = ToolProvider.getSystemJavaCompiler( ).getStandardFileManager( null, null, null ); sfm.list( new JavaFileManager.Location( ) { @Override public boolean isOutputLocation( ) { return false; } @Override public String getName( ) { return "CLASSNAME"; } }, "com", kinds, true ); countOpenFiles( ); } private void countOpenFiles( ) { final String processName = java.lang.management.ManagementFactory.getRuntimeMXBean( ).getName( ); final long pid = Long.parseLong( processName.split( "@" )[ 0 ] ); try { final Runtime rt = Runtime.getRuntime( ); final Process pr = rt.exec( "lsof -p " + pid ); int ctr = 0; final BufferedReader br = new BufferedReader( new InputStreamReader( pr.getInputStream( ) ) ); while ( ( br.readLine( ) ) != null ) ctr++; pr.waitFor( ); pr.destroy( ); System.out.println( "Open files: " + ctr ); } catch ( final Exception e ) { e.printStackTrace( ); } } Can anybody confirm this? Best, Andreas

More information about the compiler-dev mailing list