Caveats and Gotchas — pandas 0.18.1 documentation (original) (raw)

Using If/Truth Statements with pandas

pandas follows the numpy convention of raising an error when you try to convert something to a bool. This happens in a if or when using the boolean operations, and, or, or not. It is not clear what the result of

if pd.Series([False, True, False]): ...

should be. Should it be True because it’s not zero-length? False because there are False values? It is unclear, so instead, pandas raises a ValueError:

if pd.Series([False, True, False]): print("I was true") Traceback ... ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().

If you see that, you need to explicitly choose what you want to do with it (e.g., use any(), all() or empty). or, you might want to compare if the pandas object is None

if pd.Series([False, True, False]) is not None: print("I was not None") I was not None

or return if any value is True.

if pd.Series([False, True, False]).any(): print("I am any") I am any

To evaluate single-element pandas objects in a boolean context, use the method .bool():

In [1]: pd.Series([True]).bool() Out[1]: True

In [2]: pd.Series([False]).bool() Out[2]: False

In [3]: pd.DataFrame([[True]]).bool() Out[3]: True

In [4]: pd.DataFrame([[False]]).bool() Out[4]: False

Bitwise boolean

Bitwise boolean operators like == and != will return a boolean Series, which is almost always what you want anyways.

s = pd.Series(range(5)) s == 4 0 False 1 False 2 False 3 False 4 True dtype: bool

See boolean comparisons for more examples.

Using the in operator

Using the Python in operator on a Series tests for membership in the index, not membership among the values.

If this behavior is surprising, keep in mind that using in on a Python dictionary tests keys, not values, and Series are dict-like. To test for membership in the values, use the method isin():

For DataFrames, likewise, in applies to the column axis, testing for membership in the list of column names.

Integer indexing

Label-based indexing with integer axis labels is a thorny topic. It has been discussed heavily on mailing lists and among various members of the scientific Python community. In pandas, our general viewpoint is that labels matter more than integer locations. Therefore, with an integer axis index _only_label-based indexing is possible with the standard tools like .ix. The following code will generate exceptions:

s = pd.Series(range(5)) s[-1] df = pd.DataFrame(np.random.randn(5, 4)) df df.ix[-2:]

This deliberate decision was made to prevent ambiguities and subtle bugs (many users reported finding bugs when the API change was made to stop “falling back” on position-based indexing).

Label-based slicing conventions

Non-monotonic indexes require exact matches

If the index of a Series or DataFrame is monotonically increasing or decreasing, then the bounds of a label-based slice can be outside the range of the index, much like slice indexing a normal Python list. Monotonicity of an index can be tested with the is_monotonic_increasing andis_monotonic_decreasing attributes.

In [11]: df = pd.DataFrame(index=[2,3,3,4,5], columns=['data'], data=range(5))

In [12]: df.index.is_monotonic_increasing Out[12]: True

no rows 0 or 1, but still returns rows 2, 3 (both of them), and 4:

In [13]: df.loc[0:4, :] Out[13]: data 2 0 3 1 3 2 4 3

slice is are outside the index, so empty DataFrame is returned

In [14]: df.loc[13:15, :] Out[14]: Empty DataFrame Columns: [data] Index: []

On the other hand, if the index is not monotonic, then both slice bounds must be_unique_ members of the index.

In [15]: df = pd.DataFrame(index=[2,3,1,4,3,5], columns=['data'], data=range(6))

In [16]: df.index.is_monotonic_increasing Out[16]: False

OK because 2 and 4 are in the index

In [17]: df.loc[2:4, :] Out[17]: data 2 0 3 1 1 2 4 3

0 is not in the index

In [9]: df.loc[0:4, :] KeyError: 0

3 is not a unique label

In [11]: df.loc[2:3, :] KeyError: 'Cannot get right slice bound for non-unique label: 3'

Endpoints are inclusive

Compared with standard Python sequence slicing in which the slice endpoint is not inclusive, label-based slicing in pandas is inclusive. The primary reason for this is that it is often not possible to easily determine the “successor” or next element after a particular label in an index. For example, consider the following Series:

In [18]: s = pd.Series(np.random.randn(6), index=list('abcdef'))

In [19]: s Out[19]: a 1.544821 b -1.708552 c 1.545458 d -0.735738 e -0.649091 f -0.403878 dtype: float64

Suppose we wished to slice from c to e, using integers this would be

In [20]: s[2:5] Out[20]: c 1.545458 d -0.735738 e -0.649091 dtype: float64

However, if you only had c and e, determining the next element in the index can be somewhat complicated. For example, the following does not work:

A very common use case is to limit a time series to start and end at two specific dates. To enable this, we made the design design to make label-based slicing include both endpoints:

In [21]: s.ix['c':'e'] Out[21]: c 1.545458 d -0.735738 e -0.649091 dtype: float64

This is most definitely a “practicality beats purity” sort of thing, but it is something to watch out for if you expect label-based slicing to behave exactly in the way that standard Python integer slicing works.

Miscellaneous indexing gotchas

Reindex versus ix gotchas

Many users will find themselves using the ix indexing capabilities as a concise means of selecting data from a pandas object:

In [22]: df = pd.DataFrame(np.random.randn(6, 4), columns=['one', 'two', 'three', 'four'], ....: index=list('abcdef')) ....:

In [23]: df Out[23]: one two three four a -2.474932 0.975891 -0.204206 0.452707 b 3.478418 -0.591538 -0.508560 0.047946 c -0.170009 -1.615606 -0.894382 1.334681 d -0.418002 -0.690649 0.128522 0.429260 e 1.207515 -1.308877 -0.548792 -1.520879 f 1.153696 0.609378 -0.825763 0.218223

In [24]: df.ix[['b', 'c', 'e']] Out[24]: one two three four b 3.478418 -0.591538 -0.508560 0.047946 c -0.170009 -1.615606 -0.894382 1.334681 e 1.207515 -1.308877 -0.548792 -1.520879

This is, of course, completely equivalent in this case to using thereindex method:

In [25]: df.reindex(['b', 'c', 'e']) Out[25]: one two three four b 3.478418 -0.591538 -0.508560 0.047946 c -0.170009 -1.615606 -0.894382 1.334681 e 1.207515 -1.308877 -0.548792 -1.520879

Some might conclude that ix and reindex are 100% equivalent based on this. This is indeed true except in the case of integer indexing. For example, the above operation could alternately have been expressed as:

In [26]: df.ix[[1, 2, 4]] Out[26]: one two three four b 3.478418 -0.591538 -0.508560 0.047946 c -0.170009 -1.615606 -0.894382 1.334681 e 1.207515 -1.308877 -0.548792 -1.520879

If you pass [1, 2, 4] to reindex you will get another thing entirely:

In [27]: df.reindex([1, 2, 4]) Out[27]: one two three four 1 NaN NaN NaN NaN 2 NaN NaN NaN NaN 4 NaN NaN NaN NaN

So it’s important to remember that reindex is strict label indexing only. This can lead to some potentially surprising results in pathological cases where an index contains, say, both integers and strings:

In [28]: s = pd.Series([1, 2, 3], index=['a', 0, 1])

In [29]: s Out[29]: a 1 0 2 1 3 dtype: int64

In [30]: s.ix[[0, 1]] Out[30]: 0 2 1 3 dtype: int64

In [31]: s.reindex([0, 1]) Out[31]: 0 2 1 3 dtype: int64

Because the index in this case does not contain solely integers, ix falls back on integer indexing. By contrast, reindex only looks for the values passed in the index, thus finding the integers 0 and 1. While it would be possible to insert some logic to check whether a passed sequence is all contained in the index, that logic would exact a very high cost in large data sets.

Reindex potentially changes underlying Series dtype

The use of reindex_like can potentially change the dtype of a Series.

In [32]: series = pd.Series([1, 2, 3])

In [33]: x = pd.Series([True])

In [34]: x.dtype Out[34]: dtype('bool')

In [35]: x = pd.Series([True]).reindex_like(series)

In [36]: x.dtype Out[36]: dtype('O')

This is because reindex_like silently inserts NaNs and the dtypechanges accordingly. This can cause some issues when using numpy ufuncssuch as numpy.logical_and.

See the this old issue for a more detailed discussion.

Parsing Dates from Text Files

When parsing multiple text file columns into a single date column, the new date column is prepended to the data and then index_col specification is indexed off of the new set of columns rather than the original ones:

In [37]: print(open('tmp.csv').read()) KORD,19990127, 19:00:00, 18:56:00, 0.8100 KORD,19990127, 20:00:00, 19:56:00, 0.0100 KORD,19990127, 21:00:00, 20:56:00, -0.5900 KORD,19990127, 21:00:00, 21🔞00, -0.9900 KORD,19990127, 22:00:00, 21:56:00, -0.5900 KORD,19990127, 23:00:00, 22:56:00, -0.5900

In [38]: date_spec = {'nominal': [1, 2], 'actual': [1, 3]}

In [39]: df = pd.read_csv('tmp.csv', header=None, ....: parse_dates=date_spec, ....: keep_date_col=True, ....: index_col=0) ....:

index_col=0 refers to the combined column "nominal" and not the original

first column of 'KORD' strings

In [40]: df Out[40]: actual 0 1 2 3
nominal
1999-01-27 19:00:00 1999-01-27 18:56:00 KORD 19990127 19:00:00 18:56:00
1999-01-27 20:00:00 1999-01-27 19:56:00 KORD 19990127 20:00:00 19:56:00
1999-01-27 21:00:00 1999-01-27 20:56:00 KORD 19990127 21:00:00 20:56:00
1999-01-27 21:00:00 1999-01-27 21🔞00 KORD 19990127 21:00:00 21🔞00
1999-01-27 22:00:00 1999-01-27 21:56:00 KORD 19990127 22:00:00 21:56:00
1999-01-27 23:00:00 1999-01-27 22:56:00 KORD 19990127 23:00:00 22:56:00 

                    4

nominal
1999-01-27 19:00:00 0.81
1999-01-27 20:00:00 0.01
1999-01-27 21:00:00 -0.59
1999-01-27 21:00:00 -0.99
1999-01-27 22:00:00 -0.59
1999-01-27 23:00:00 -0.59

Differences with NumPy

For Series and DataFrame objects, var normalizes by N-1 to produce unbiased estimates of the sample variance, while NumPy’s var normalizes by N, which measures the variance of the sample. Note that covnormalizes by N-1 in both pandas and NumPy.

Thread-safety

As of pandas 0.11, pandas is not 100% thread safe. The known issues relate to the DataFrame.copy method. If you are doing a lot of copying of DataFrame objects shared among threads, we recommend holding locks inside the threads where the data copying occurs.

See this linkfor more information.

HTML Table Parsing

There are some versioning issues surrounding the libraries that are used to parse HTML tables in the top-level pandas io function read_html.

Issues with lxml

Issues with BeautifulSoup4 using lxml as a backend

Issues with BeautifulSoup4 using html5lib as a backend

Issues with using Anaconda

Note

Unless you have both:

  • A strong restriction on the upper bound of the runtime of some code that incorporates read_html()
  • Complete knowledge that the HTML you will be parsing will be 100% valid at all times

then you should install html5lib and things will work swimmingly without you having to muck around with conda. If you want the best of both worlds then install both html5lib and lxml. If you do installlxml then you need to perform the following commands to ensure that lxml will work correctly:

remove the included version

conda remove lxml

install the latest version of lxml

pip install 'git+git://github.com/lxml/lxml.git'

install the latest version of beautifulsoup4

pip install 'bzr+lp:beautifulsoup'

Note that you need bzr and git installed to perform the last two operations.

Byte-Ordering Issues

Occasionally you may have to deal with data that were created on a machine with a different byte order than the one on which you are running Python. A common symptom of this issue is an error like

Traceback ... ValueError: Big-endian buffer not supported on little-endian compiler

To deal with this issue you should convert the underlying NumPy array to the native system byte order before passing it to Series/DataFrame/Panel constructors using something similar to the following:

In [41]: x = np.array(list(range(10)), '>i4') # big endian

In [42]: newx = x.byteswap().newbyteorder() # force native byteorder

In [43]: s = pd.Series(newx)

See the NumPy documentation on byte order for more details.