TST/CI: Use moto to mock S3 calls · Issue #17325 · pandas-dev/pandas (original) (raw)

Skip to content

Navigation Menu

• • GitHub Copilot Write better code with AI
  • GitHub Advanced Security Find and fix vulnerabilities
  • Actions Automate any workflow
  • Codespaces Instant dev environments
  • Issues Plan and track work
  • Code Review Manage code changes
  • Discussions Collaborate outside of code
  • Code Search Find more, search less
• Explore
  • Learning Pathways
  • Events & Webinars
  • Ebooks & Whitepapers
  • Customer Stories
  • Partners
  • Executive Insights
• • GitHub Sponsors Fund open source developers
  • The ReadME Project GitHub community articles
• • Enterprise platform AI-powered developer platform
• Pricing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Appearance settings

Description

https://github.com/spulec/moto

I think most of our tests hitting s3 can be mocked now that we're using s3fs and boto3. We don't have any code doing network requests, auth, etc. As an example:

import pytest
import pandas as pd
from moto import mock_s3
import boto3

f = open("pandas/tests/io/data/tips.csv", 'rb')

@mock_s3
def test_foo():
    conn = boto3.resource("s3", region_name="us-east-1")
    conn.create_bucket(Bucket="pandas-test")

    conn.Bucket("pandas-test").put_object(Key="not-a-real-key.csv", Body=f)
    result = pd.read_csv("s3://pandas-test/not-a-real-key.csv")
    assert isinstance(result, pd.DataFrame)
    with pytest.raises(Exception):
        result = pd.read_csv("s3://pandas-test/for-real-not-a-real-key.csv")

It'd probably be good to have a few integration tests that actually hit S3, but I think most can be replaced with mocked calls.