Ability to catch sqlite3 error log (original) (raw)

August 30, 2023, 3:47pm 1

sqlite3 has a callback facility wherein the engine can call into the application to report error & diagnostic conditions with a numeric/text message. sqlite authors promote using this hook to C/C++ developers as supplying valuable information for understanding errors (and some performance optimization decisions!). However, the cpython _sqlite module does not expose this function. So python sqlite3 users cannot take advantage.

It would be great if, similar to the sqlite3 .set_trace_callback(), a similar callback existed for the error log. One complication is that this callback is documented to be as restricted in context as a signal handler, so presumably some queuing / mapping would be necessary.

NB: This is not equivalent to existing error RC’s from sqlite3 functions, which are already mapped to exceptions, but rather akin to logging records.

https://www.sqlite.org/errlog.html

AA-Turner (Adam Turner) August 30, 2023, 4:29pm 2

erlendaasland (Erlend E. Aasland) August 30, 2023, 10:09pm 3

The problem with the log callback, is that you have to register it with sqlite3_config() which must be called before SQLite initialisation. Currently, SQLite is initialised during module init (import sqlite3), so it is not trivial to expose such functionality. One option could be to not initialise SQLite during module init, and instead initialise lazily, but we’d still have to be sure that no database connections are open and no SQLite APIs are in use, so it has a very rigid contract with the user.

Note that the other callbacks (trace, authorizer, progress) work with database handles, so they do not have the restrictions of sqlite3_config().

erlendaasland (Erlend E. Aasland) August 30, 2023, 10:14pm 4

Also notice that even though the SQLITE_CONFIG_LOG config option is currently marked as an “anytime configuration option”, that feature is not guaranteed. Quoting the SQLite docs:

The set of anytime configuration options can change (by insertions and/or deletions) from one release of SQLite to the next.

New configuration options may be added in future releases of SQLite. Existing configuration options might be discontinued.

fche (Frank Ch. Eigler) August 31, 2023, 7:54pm 5

Even with that limitation (“anytime” but who knows whether forever), this would be useful. Since sqlite3 does report errors if such calls are mistimed (change from “anytime”), the python wrapper could turn the call into an exception.

erlendaasland (Erlend E. Aasland) August 31, 2023, 8:18pm 6

I suggest you check out the apsw[1] module; it has built-in support for the SQLite error log mechanism:

https://rogerbinns.github.io/apsw/tips.html#diagnostics-tips

  1. Another Python SQLite Wrapper ↩︎

storchaka (Serhiy Storchaka) September 1, 2023, 4:44am 7

It is not such large problem. You can always register a callback which does nothing by default. But if you then register a Python callable, it will call that callable.

The question is how much an empty log callback affects performance?

erlendaasland (Erlend E. Aasland) September 1, 2023, 5:18am 8

That’s a possibility.

I’ll create a proof-of-concept branch, so we can benchmark it.

erlendaasland (Erlend E. Aasland) September 1, 2023, 8:40am 9

fche (Frank Ch. Eigler) September 1, 2023, 8:00pm 10

Thanks for the reference. A different flavour (requiring quite a bit of code porting) - and fast moving (so more version conditionals).

erlendaasland (Erlend E. Aasland) September 1, 2023, 8:22pm 11

Definitely. apsw is free to do whatever they want. sqlite3 OTOH is bound by rigid backwards compatibility requirements and the DB API, so it is a lot less flexible. That does not mean we’re opposed to improve it, or (slowly) add new features, though :slight_smile:

FWIW, I think apsw is a very good piece of softwre.