Should values for LIBSQLITE3_LIBS be prefixed by "-l" or not? (original) (raw)
Im trying to build cpython, against a home built version of sqlite. I’ve spent all day, but can’t make it work. Please help me understand this first bit.
../configure \
LIBSQLITE3_CFLAGS=-I~/b/sqlite/build-3.40 \
LIBSQLITE3_LIBS=/home/velle/b/sqlite/build-3.40/.libs/libsqlite3.so
The command above does something right. For all other values I have tried, configure says sqlite is missing.
For the values above it looks like this, in config.status.
"MODULE__SQLITE3_STATE=yes\n"\
"MODULE__SQLITE3_CFLAGS=-I~/b/sqlite/build-3.40 -I$(srcdir)/Modules/_sqlite\n"\
"MODULE__SQLITE3_LDFLAGS=/home/velle/b/sqlite/build-3.40/.libs/libsqlite3.so\n"\
But notice the last one, it’s not called LIBS, its called LDFLAGS, and I would expect the value to be prefixed by -l. So I looked at the config.status for a standard build, an in that case yes, the path is prefixed by -l, and looks like this.
"MODULE__SQLITE3_STATE=yes\n"\
"MODULE__SQLITE3_CFLAGS= -I$(srcdir)/Modules/_sqlite\n"\
"MODULE__SQLITE3_LDFLAGS=-lsqlite3\n"\
So should I use “-l” or not for LIBSQLITE3_LIBS?
If I run make, there are no fatal errors, but at the very end it says.
The necessary bits to build these optional modules were not found:
_dbm _gdbm _tkinter
To find the necessary bits, look in configure.ac and config.log.
Is there a problem with configure? Or am I using it wrong? Any help is appreciated 
Details:
cpython, source, branch 3.13
sqlite, source, branch 3.40.2
Ubuntu 22.04
nad (Ned Deily) July 6, 2024, 9:35pm 2
The values you are using for the sqllite3 include file and library directories look quite unusual for a Unix build, so, without any further info on how you configured and built SQLite3, we can only speculate. But if you try the following two file commands from a shell, you can have some confidence that these are the right directory paths:
$ file "/home/velle/b/sqlite/build-master/include/sqlite3.h"
/home/velle/b/sqlite/build-master/include/sqlite3.h: c program text, ASCII text
$ file "/home/velle/b/sqlite/build-3.40/.libs/libsqlite3.so"
file /home/velle/b/sqlite/build-3.40/.libs/libsqlite3.so [..] shared library [..]
And, if your results are similar, try using something like this in your cpython configure:
LIBSQLITE3_CFLAGS="-I/home/velle/b/sqlite/build-master" \
LIBSQLITE3_LIBS="-L/home/velle/b/sqlite/build-3.40/.libs -lsqlite3"
In other words, the LIBSQLITE3_CFLAGS parameter should have an -I value that points to the directory containing the sqlite3 include files and the LIBSQLITE3_LIBS parameter should have a -L value that points to the directory containing the sqlite shared library with a name starting with libsqlite3.so.
(Also, this question would be better asked in the Python Help category; it doesn’t really have anything to do with the C API. Thanks!)
ThomasW (Thomas Wessel) July 6, 2024, 9:55pm 3
Ooops. There was a typo in my original post, which I have now corrected. sqlite/build-master → sqlite/build-3.40.
Below I will try to share how I built sqlite.
cd ~/b
git clone git@github.com:sqlite/sqlite.git
cd sqlite
git checkout origin/branch-3.40
mkdir build-3.40; cd build-3.40
../configure
make
There is no include folder in the build folder (or anywhere else) for sqlite. The header files are directly in the build-3.40 folder.
See build-3.40 tree content in this gist: gist:042c9bd6213dce4df7dd254b6e7d38b2 · GitHub
nad (Ned Deily) July 6, 2024, 9:55pm 4
One more thing that I forgot to mention. If you are linking to a shared library installed in a non-standard directory location, you may need to supply more information at run time for the system’s dynamic loader to find it. For example, in this case, one way to do that might be to set the LD_LIBRARY_PATH environment variable. This StackExchange post has more information on this somewhat complex topic.
nad (Ned Deily) July 6, 2024, 10:04pm 5
It looks like you skipped the make install step when building SQLite and are trying to link directly from an SQLite build directory. That is usually not a good idea. Pick a location where you want to install to and specify it with a --prefix=/path/to/installdirectory on the SQLite configure, add a make install following the make, and then use that location in the cpython configure with something like:
LIBSQLITE3_CFLAGS="-I/path/to/installdirectory/include" \
LIBSQLITE3_LIBS="-L/path/to/installdirectory/lib -lsqlite3"
nad (Ned Deily) July 6, 2024, 10:09pm 6
Also, you don’t say why you are building your own SQLite but it would be much easier if you could use the Ubuntu-supplied version. The Python Developer’s Guide has information on using system-supplied dependencies.
ThomasW (Thomas Wessel) July 6, 2024, 10:15pm 7
Yes it would be much easier to use that one, and I would have loved to But I need Python with >= sqlite3.40. Reason here: SQLite Forum: specifying column type for a view.
(I’m trying out your suggestions at the moment)
ThomasW (Thomas Wessel) July 6, 2024, 10:23pm 8
I deliberately chose not to call make install. I thought it was only used for installing system wide. I have had luck doing this before, without make install, howeer not with python/sqlite.
Here is what I just did:
cd /home/velle/sqlite-3.40
rm * -rf
../configure --prefix=/home/velle/sqlite-3.40
make
make install
And here is the content of the install destination:
$ tree ~/sqlite-3.40
/home/velle/sqlite-3.40
├── bin
│ └── sqlite3
├── include
│ ├── sqlite3ext.h
│ └── sqlite3.h
└── lib
├── libsqlite3.a
├── libsqlite3.la
├── libsqlite3.so -> libsqlite3.so.0.8.6
├── libsqlite3.so.0 -> libsqlite3.so.0.8.6
├── libsqlite3.so.0.8.6
└── pkgconfig
└── sqlite3.pc
Will try compiling cpython now.
ThomasW (Thomas Wessel) July 6, 2024, 11:05pm 9
I managed to build cpython, all the way through. But when inside this custom python, it still uses sqlite 3.37, which is the default version on Ubuntu.
I am too tired to continue now. I already made tons of minor errors and types the last hour. But I really appreciate your help!! 
I will keep trying tomorrow
ThomasW (Thomas Wessel) July 7, 2024, 9:19am 10
It works!!! Thanks a lot for your help Ned 
Looking at it now, its crazy how that took me more than a day.
Summary below (also with new locations for installations).
Building sqlite 3.40
Build/install:
cd ~/b
git clone git@github.com:sqlite/sqlite.git
cd sqlite
git checkout origin/branch-3.40
mkdir build-3.40; cd build-3.40
../configure --prefix=/opt/sqlite-3.40
make
sudo make install
Test:
$ /opt/sqlite-3.40/bin/sqlite3 --version
3.40.2 2023-01-14 19:36:47 c66c7734feceb6ee9ae079df4fa4d078891d9e5fe6a44a2acfc482598a7176b8
Building python with sqlite 3.40
Build/install:
cd ~/b
git clone git@github.com:python/cpython.git
git checkout origin/3.13
cd cpython
mkdir build-custom; cd build-custom
../configure \
--prefix=/opt/python-3.13 \
LIBSQLITE3_CFLAGS=-I~/sqlite-3.40/include \
LIBSQLITE3_LIBS="-L~/sqlite-3.40/lib -lsqlite3"
make
sudo make install
Test
Remember to set LD_LIBRARY_PATH !!
$ /opt/python-3.13/bin/python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
3.37.2
$ LD_LIBRARY_PATH=/opt/sqlite-3.40/lib /opt/python-3.13/bin/python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
4.40.2
Winand (Makarov Andrey) October 16, 2024, 2:17pm 11
Setting up LIBSQLITE3_CFLAGS and LIBSQLITE3_LIBS didn’t help me. sqlite3 extension module is built successfully but it fails to be imported so it’s excluded from the installation. I think you didn’t have that issue because you have some version of SQLite installed in your system. And then in runtime you switch sqlite version by changing LD_LIBRARY_PATH variable.
But the right way to fix this is to specify runtime library directory (rpath) during configure stage:
./configure LDFLAGS=-Wl,-rpath=$SQLITE_INSTALL_PATH/lib PKG_CONFIG_PATH=$SQLITE_INSTALL_PATH/lib/pkgconfig
Also I set up PKG_CONFIG_PATH instead of LIBSQLITE3_CFLAGS, LIBSQLITE3_LIBS. But I think it doesn’t matter.
Here’s my full build script python-centos7-docker-build/entrypoint.sh at master · Winand/python-centos7-docker-build · GitHub