ENH: DtypeWarning message enhancement · Issue #58174 · pandas-dev/pandas (original) (raw)

Feature Type

• Adding new functionality to pandas
• Changing existing functionality in pandas
• Removing existing functionality in pandas

Problem Description

The current DtypeWarning message:
DtypeWarning: Columns (15,19) have mixed types. Specify dtype option on import or set low_memory=False.

Having a column name more than a column number would make sense.

Preferred warning message:

1. DtypeWarning: Columns (15: Comment, 19: Description) have mixed types. Specify dtype option on import or set low_memory=False.
2. DtypeWarning: Columns (15, 19) have mixed types. Specify dtype option ({'Comment': str, 'Description': str}) on import or set low_memory=False.

So the next user action can be easy:

read_csv(..., dtype={"Comment": str, "Description": str})

Feature Description

Current code:

def _concatenate_chunks(chunks: list[dict[int, ArrayLike]]) -> dict: """ Concatenate chunks of data read with low_memory=True.

The tricky part is handling Categoricals, where different chunks
may have different inferred categories.
"""
names = list(chunks[0].keys())
warning_columns = []

result: dict = {}
for name in names:
    arrs = [chunk.pop(name) for chunk in chunks]
    # Check each arr for consistent types.
    dtypes = {a.dtype for a in arrs}
    non_cat_dtypes = {x for x in dtypes if not isinstance(x, CategoricalDtype)}

    dtype = dtypes.pop()
    if isinstance(dtype, CategoricalDtype):
        result[name] = union_categoricals(arrs, sort_categories=False)
    else:
        result[name] = concat_compat(arrs)
        if len(non_cat_dtypes) > 1 and result[name].dtype == np.dtype(object):
            warning_columns.append(str(name))

if warning_columns:
    warning_names = ",".join(warning_columns)
    warning_message = " ".join(
        [
            f"Columns ({warning_names}) have mixed types. "
            f"Specify dtype option on import or set low_memory=False."
        ]
    )
    warnings.warn(warning_message, DtypeWarning, stacklevel=find_stack_level())
return result
```

Alternative Solutions

Replace columns number with column names

Additional Context

No response