Use directory of Markdown as database? (original) (raw)

Has anyone had success at loading a directory (of directories) of markdown files, and querying it using Python, similar to SQLite? I saw there is MarkdownDB (Javascript) - curious what exists in the Python world, if there are libraries like that.

Had an idea for a knowledge base, where you could query the metadata, select random pages given a tag, etc. And easily add knowledge by just saving markdown as you go…

Like MongoDB… but plain-jane files…

elis.byberi (Elis Byberi) May 6, 2025, 1:03am 2

There are plenty of libraries that can analyze Markdown files, convert them to other formats (e.g., JSON), or populate a SQLite database using data from Markdown, and so on.

Once you convert Markdown into a format supported by databases, you can take full advantage of query languages. I don’t see a strong reason to query data directly from a Markdown file.

jro (Jonathan Rogivue) May 6, 2025, 2:02am 3

Can you share recommendations?

What I’m intending is “human friendly” version of an SQLite database-as-file, mixed somewhat like MongoDB’s document approach. Instead of needing a DB client to interact with the data, you’d just drop in a markdown file into the filesystem and can query its contents, header metadata (including “tags: ”), and perhaps the file structure (example querying “piano” could find: /music/piano/songs/coffee-cold.md) - then in Python I’d have access to the file contents thru an ORM-like interface.

Importing the markdown files into SQLite is definitely an option, but I’m hoping there’s already a library out there which could help out.

csm10495 (Charles Machalow) May 7, 2025, 12:10am 4

We did something like this at a previous job. The markdown files lived in a repo and building the repo made static html files that were served by a managed IIS instance. We used https://www.mkdocs.org/ to build the site.

Basically you can use a directory structure along with certain tags inside the markdown files or just the name of the markdown file itself.

One of the reasons for centralized databases is to have a single source of truth (and ACID properties). If you don’t have those requirements, then yeah a folder of markdown files can work fine.

All of this kind of depends more on the requirements. Do the files need to be served? Do people have local clones? Do you want people to be able to add files remotely?

All of this can be done, but the basics are there.