CodeParser not opening source files with proper decoder · Issue #107 · nedbat/coveragepy (original) (raw)

Skip to content

Navigation Menu

• • GitHub Copilot Write better code with AI
  • GitHub Advanced Security Find and fix vulnerabilities
  • Actions Automate any workflow
  • Codespaces Instant dev environments
  • Issues Plan and track work
  • Code Review Manage code changes
  • Discussions Collaborate outside of code
  • Code Search Find more, search less
• Explore
  • Learning Pathways
  • Events & Webinars
  • Ebooks & Whitepapers
  • Customer Stories
  • Partners
  • Executive Insights
• • GitHub Sponsors Fund open source developers
  • The ReadME Project GitHub community articles
• • Enterprise platform AI-powered developer platform
• Pricing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Description

Originally reported by Brett Cannon (Bitbucket: brettcannon, GitHub: brettcannon)

In CodeParser.init() you will notice that it is opening a source file and then reading it, relying on the default encoding for open(). This can trigger a UnicodeDecodeError if the source file specifies an explicit encoding other than Unicode (on Python 3).

For example, in Python's stdlib, Lib/sqlite3/test/dbapi.py has a specified encoding of ISO-8859-1. But because the CodeParser doesn't use something like tokenize.detect_encoding() (http://docs.python.org/py3k/library/tokenize.html#tokenize.detect_encoding) the read fails as there is some bytes in there not allowed under UTF-8 but are valid under ISO-8859-1.

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/107
• This issue had attachments: tokenize_open.diff. See the original issue for details.