pandas (original) (raw)

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample, a copy-pastable example

import pandas as pd pd.DataFrame([[pd.Timestamp("2021-01-01")]], index=[1, 2, 3])

This will throw an exception:

Traceback (most recent call last):
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 1770, in create_block_manager_from_arrays
    mgr = BlockManager(blocks, axes)
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 910, in __init__
    self._verify_integrity()
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 917, in _verify_integrity
    raise construction_error(tot_items, block.shape[1:], self.axes)
ValueError: Shape of passed values is (1, 1), indices imply (3, 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/frame.py", line 702, in __init__
    mgr = arrays_to_mgr(
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 135, in arrays_to_mgr
    return create_block_manager_from_arrays(
  File "/tmp/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 1772, in create_block_manager_from_arrays
    raise construction_error(len(arrays), arrays[0].shape, axes, e)
ValueError: Shape of passed values is (1, 1), indices imply (3, 1)

The same exception will also be raised when the data argument is a list-of-dicts but not when data is a dict-of-lists:

pd.DataFrame([{"x": pd.Timestamp("2021-01-01"}], index=[1, 2, 3]) # this will fail due to the same exception pd.DataFrame({"x": [pd.Timestamp("2021-01-01")]}, index=[1, 2, 3]) # this format is ok

Problem description

From what I can tell, the DataFrame constructor supports supplying a length-1 list and an index of length greater than 1 as long as the list doesn't contain a value of type Timestamp / datetime64. Raising an error only for these types is therefore inconsistent and could lead to user confusion. It is also a change from the behaviour in pandas 1.2 where it was possible to supply a length-1 list containing a Timestamp.

>>> pd.__version__
'1.3.1'
>>> pd.DataFrame([[7]], index=[1, 2, 3])
   0
1  7
2  7
3  7
>>> pd.DataFrame([["abc"]], index=[1, 2, 3])
     0
1  abc
2  abc
3  abc
>>> pd.DataFrame([[object()]], index=[1, 2, 3])
                                0
1  <object object at 0x111a16e20>
2  <object object at 0x111a16e20>
3  <object object at 0x111a16e20>

In Pandas 1.2.5:

>>> import pandas as pd
>>> pd.__version__
'1.2.5'
>>> pd.DataFrame([[pd.Timestamp("2021-01-01")]], index=[1, 2, 3])
           0
1 2021-01-01
2 2021-01-01
3 2021-01-01

Expected Output

The length-1 list should be repeated to the length of the index:

>>> pd.DataFrame([[pd.Timestamp("2021-01-01")]], index=[1, 2, 3])
           0
1 2021-01-01
2 2021-01-01
3 2021-01-01

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit : c7f7443
python : 3.9.4.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Mon May 3 20:41:19 PDT 2021; root:xnu-4903.278.68~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 1.3.1
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.2
pip : 20.2.3
setuptools : 49.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

Notes

This appear to be caused by the code going through the elif isinstance(data, ABCExtensionArray) branch of sanitize_array which causes it to return early before _sanitize_ndim is called. Removing that early return appears to fix the problem and it didn't cause any tests to fail when I ran ./test_fast.sh. If this change makes sense I'd be happy to put up a PR.