Issue 25655: Python errors related to failures loading DLL's lack information (original) (raw)

Created on 2015-11-18 08:37 by never-eat-yellow-snow, last changed 2022-04-11 14:58 by admin.

Messages (16)
msg254838 - (view)	Author: (never-eat-yellow-snow)	Date: 2015-11-18 08:37
Currently you get errors like this: ImportError: DLL load failed: The specified procedure could not be found. ImportError: DLL load failed: The specified module could not be found. It would be nice to include more information, at least the name of the dll which could not be loaded. Maybe also the name of the (dependent) dll which could not be found. Currently, I use ProcessMonitor to debug which dll could not be found, because the error message is lacking important information. Note: I tagged the two versions I use, but probably all python versions are affected by this issue.
msg254849 - (view)	Author: Steve Dower (steve.dower) *	Date: 2015-11-18 16:56
I don't know that we can necessarily provide correct information for those errors as it depends where they actually fail. If we're simply passing on an error from the loader, then there's very little we can do. We may be able to make an educated guess based on our context, but there's a fairly high chance we'll guess wrong in some cases and make them more difficult to debug (unless you know that the error is only a guess, in which case you're back to procmon or Dependency Walker). You're right that all versions are affected, however I think improving these diagnostics is only within scope for 3.5 and 3.6 at this stage.
msg377326 - (view)	Author: Philippe Ombredanne (pombredanne) *	Date: 2020-09-22 14:18
From https://bugs.python.org/issue41836 closed as a dupe of this: When the dependency of a DLL is missing (at least on Windows) the error " OSError: [WinError 126] The specified module could not be found" is raised when calling ctypes.CDLL(dll_path) even when this "dll_path" exists... because the error comes from another DLL. These errors are really hard to diagnose because the path of the missing DLL is not returned in the exception message. Returning it would help fixing these kind of errors quickly. Researching errors such as this one https://github.com/nexB/scancode-toolkit/issues/2236 wastes quite a bit of time and would be made a non issue if we had the path in the error message. and this reply from Eric Smith: https://bugs.python.org/msg377324 > Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-09-22 14:13 > My understanding is that Windows doesn't tell you which DLL is missing. I think the best we could do is append something to the error message saying "or one its dependencies".
msg377327 - (view)	Author: Philippe Ombredanne (pombredanne) *	Date: 2020-09-22 14:26
Eric Smith, you wrote: > My understanding is that Windows doesn't tell you which DLL is missing. I think the best we could do is append something to the error message saying "or one its dependencies". If we have such an error message, this means the main DLL exists: the original path passed to ctypes exists and is a valid DLL otherwise the message would be different. So I think that this is always a direct or indirect dependency of that primary DLL that would be missing and we could be explicit in the error message. We could also provide some hints in the error message on how to research the issue may be?
msg377335 - (view)	Author: Eryk Sun (eryksun) *	Date: 2020-09-22 15:06
> " OSError: [WinError 126] The specified module could not be found" is > raised when calling ctypes.CDLL(dll_path) even when this "dll_path" > exists... because the error comes from another DLL. That's the old error. bpo-36085 changed it to FileNotFoundError, with the message "Could not find module '%.500S'. Try using the full path with constructor syntax." bpo-39393 modified the message to "Could not find module '%.500S' (or one of its dependencies). Try using the full path with constructor syntax." IMO, the most direct way to resolve the problem is by enabling "loader snaps" for python.exe via gflags and attaching a native debugger to the process. The loader outputs debug strings that show the computed DLL search path (from LdrpComputeLazyDllPath), each attempt to resolve the dependent DLL to a directory in the search path (via LdrpResolveDllName), and the final result from loader's work queue (from LdrpProcessWork), which includes the dependent DLL that caused loading to fail and the parent module (DLL or EXE) that depends on it.
msg377351 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-09-22 22:46
> IMO, the most direct way to resolve the problem is by enabling "loader snaps" for python.exe via gflags and attaching a native debugger to the process ... This is indeed the best way to go about solving it, so you can see why we don't put it in an error message or take responsibility for documenting the process. It's not for the faint-hearted :) Also, the recommended releases of WinDBG (from the Microsoft Store) no longer include gflags, though I believe once you're in the debugger it will just break at the point where the DLL can't be loaded and it's "simple" to get its expected name. I wouldn't refuse a docs PR to add a short section pointing to this page and explaining its relevance: https://docs.microsoft.com/cpp/build/reference/dependents I *would* stop short of writing a whole tutorial on how to do it. That's a great topic for someone's blog, and will likely get better SEO and social attention from not being in the docs.
msg377360 - (view)	Author: Philippe Ombredanne (pombredanne) *	Date: 2020-09-23 07:37
> I wouldn't refuse a docs PR to add a short section pointing to > this page and explaining its relevance: > https://docs.microsoft.com/cpp/build/reference/dependents Steve, would you see this as a note in https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries What about something like this? class ctypes.CDLL ..... Note: On Windows a call to CDLL(name) may fail even if the DLL name exists when a dependent DLL of this DLL is found. This will lead to an OSErrror error with the message "[WinError 126] The specified module could not be found". This error message does not contains the name of the missing DLL because the Windows API does not return this information making this error hard to diagnose. To resolve this error and determine which DLL is missing, you need to find the list of dependent DLLs using Windows debugging and tracing tools. See https://docs.microsoft.com/cpp/build/reference/dependents for some explanations.
msg377400 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-09-23 15:46
> would you see this as a note in https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries Haven't looked at the PR, but it probably needs to be somewhere in the import docs as well, to do with native extension modules. That's where most people run into this. And in general the solution is either going to involve moving/renaming files or calling os.add_dll_directory, so a link to the latter may also be useful. I think we've got a Sphinx tag for platform-specific information? If we do, it should use that. (Unless I'm just thinking of the "API availability" tag rather than a "Note" style box.)
msg377416 - (view)	Author: Philippe Ombredanne (pombredanne) *	Date: 2020-09-23 18:27
So the other locations to add docs would be petinetially - https://docs.python.org/3/library/os.html?#os.add_dll_directory - https://docs.python.org/3/extending/windows.html - https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll - https://docs.python.org/3/using/windows.html#finding-modules Which ones would be the best? Also AFAIK there is no Windows Sphinx tag beyond .. availability::
msg377418 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-09-23 19:33
Thanks for doing the search :) > - https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll Probably not here. > - https://docs.python.org/3/using/windows.html#finding-modules Perhaps it is best to put a new section here like what you posted above (but more generic for ctypes and imports), and then link to it from the other places? > - https://docs.python.org/3/library/os.html?#os.add_dll_directory e.g. "This function may be used to work around errors" > - https://docs.python.org/3/extending/windows.html e.g. "If your extension module relies on any DLLs other than those included with Windows or CPython, you will need to include them or else users may receive 'module not found' errors. See for more details." (Some of that text may already be there, been a while since I read that one.) > Also AFAIK there is no Windows Sphinx tag beyond .. availability:: Yeah, I think I was thinking of a different project. But if it's all in Windows-specific sections anyway, and pointing towards the Windows doc page, then it won't matter.
msg379301 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-10-22 15:39
New changeset b6f2fc90409e291822166d74ce7402e0ef4dba91 by Philippe Ombredanne in branch 'master': bpo-25655: Improve Win DLL loading failures doc (GH-22372) https://github.com/python/cpython/commit/b6f2fc90409e291822166d74ce7402e0ef4dba91
msg379303 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-10-22 15:41
Thanks for the PR!
msg379306 - (view)	Author: miss-islington (miss-islington)	Date: 2020-10-22 15:48
New changeset 5d8bc65ba5be5742b3a4cc470dfd990512bdaa93 by Miss Skeleton (bot) in branch '3.8': bpo-25655: Improve Win DLL loading failures doc (GH-22372) https://github.com/python/cpython/commit/5d8bc65ba5be5742b3a4cc470dfd990512bdaa93
msg379307 - (view)	Author: Eryk Sun (eryksun) *	Date: 2020-10-22 15:50
Steve, the PR that you pushed has the wrong error and error message. I told Philippe in that ctypes raises FileNotFoundError with no error code. For example: >>> try: ctypes.CDLL('spam') ... except OSError as e: err = e ... >>> err FileNotFoundError("Could not find module 'spam' (or one of its dependencies). Try using the full path with constructor syntax.") >>> err.winerror is None True The advice to use dumpbin is fine and works well in simple cases. I wouldn't use it generally since recursiveley parsing through the dependency graph of every dependent DLL could be tedious.
msg379308 - (view)	Author: miss-islington (miss-islington)	Date: 2020-10-22 16:02
New changeset f22f874a66d6ddb32fa74ad4325199db7e4c25fd by Miss Skeleton (bot) in branch '3.9': bpo-25655: Improve Win DLL loading failures doc (GH-22372) https://github.com/python/cpython/commit/f22f874a66d6ddb32fa74ad4325199db7e4c25fd
msg379315 - (view)	Author: Steve Dower (steve.dower) *	Date: 2020-10-22 16:40
> Steve, the PR that you pushed has the wrong error and error message. Ah whoops. I'll reopen and hopefully someone can fix it up. > The advice to use dumpbin is fine and works well in simple cases. It's only meant as a starting point. It's possible to brute force your way through more complex cases, but to describe anything more advanced we'd be writing a tutorial in our docs, which is not appropriate.