[SPARK-43563][SPARK-43459][SPARK-43451][SPARK-43506] Remove squeeze from read_csv & enabling more tests. by itholic · Pull Request #42551 · apache/spark (original) (raw)

What changes were proposed in this pull request?

This PR proposes to remove squeeze parameter from read_csv to follow the behavior of latest pandas. See pandas-dev/pandas#40413 and pandas-dev/pandas#43427 for detail.

This PR also enables more tests for pandas 2.0.0 and above.

Why are the changes needed?

To follow the behavior of latest pandas, and increase the test coverage.

Does this PR introduce any user-facing change?

squeeze will be no longer available from read_csv. Otherwise, it's test-only.

How was this patch tested?

Enabling & updating the existing tests.