Fetch data for 15k PyPI packages by hugovk · Pull Request #44 · hugovk/top-pypi-packages (original) (raw)

The attempt to fetch the data for all PyPI packages (#41) was unsuccessful. From the logs:

generate.sh: line 26: 575013 Killed                  /home/botuser/.local/bin/pypinfo --all --json --indent 0 --limit 10000000 --days 29 "" project > top-pypi-packages-30-days-all.json

This had worked locally when testing with free quota. Perhaps it was a memory problem on the server. Anyway, let's go down from ~600k projects to a hopefully more manageable 15k. We can try and increase in later months.

And there's a long tail in PyPI downloads (graphs from 10 Nov 2024), so the top end will be more useful:

image

Same thing with log scale:

image