Update monthly, takes too much quota to do fortnightly by hugovk · Pull Request #9 · hugovk/top-pypi-packages (original) (raw)

It now takes too much to do fortnightly. From the logs:

Fri Jan 31 17:30:01 EET 2020
From github.com:hugovk/top-pypi-packages
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Traceback (most recent call last):
  File "/usr/local/bin/pypinfo", line 11, in <module>
    sys.exit(pypinfo())
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pypinfo/cli.py", line 150, in pypinfo
    query_rows = query_job.result(timeout=timeout // 1000)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/job.py", line 1932, in result
    super(QueryJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/job.py", line 528, in result
    return super(_AsyncJob, self).result(timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/future/polling.py", line 106, in result
    self._blocking_poll(timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/job.py", line 1906, in _blocking_poll
    super(QueryJob, self)._blocking_poll(timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/future/polling.py", line 85, in _blocking_poll
    retry_(self._done_or_raise)()
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/future/polling.py", line 62, in _done_or_raise
    if not self.done():
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/job.py", line 1894, in done
    project=self.project, timeout_ms=timeout_ms)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/client.py", line 523, in _get_query_results
    retry, method='GET', path=path, query_params=extra_params)
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/bigquery/client.py", line 275, in _call_api
    return call()
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/lib/python3.6/dist-packages/google/cloud/_http.py", line 293, in api_request
    raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://www.googleapis.com/bigquery/v2/projects/top-pypi-packages/queries/9098e94d-b383-46fc-862a-558234ec8e7c?maxResults=0&timeoutMs=10000: Quota exceeded: Your project exceeded quota for free query bytes scanned. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors

First of the month

30 17 1 * * ( eval "$(ssh-agent -s)"; ssh-add ~/.ssh/id_rsa-top-pypi-packages; /home/botuser/github/top-pypi-packages/top-pypi-packages$

If it starts taking too much quota for monthly, maybe it'll be time to drop the top-365 data, or update that less frequently (annually?).