Version 0.12.0 (July 24, 2013) — pandas 2.2.3 documentation (original) (raw)

This is a major release from 0.11.0 and includes several new features and enhancements along with a large number of bug fixes.

Highlights include a consistent I/O API naming scheme, routines to read html, write MultiIndexes to csv files, read & write STATA data files, read & write JSON format files, Python 3 support for HDFStore, filtering of groupby expressions via filter, and a revamped replace routine that accepts regular expressions.

API changes#

• The I/O API is now much more consistent with a set of top level reader functions accessed like pd.read_csv() that generally return a pandas object.
  • read_csv
  • read_excel
  • read_hdf
  • read_sql
  • read_json
  • read_html
  • read_stata
  • read_clipboard
    The corresponding writer functions are object methods that are accessed like df.to_csv()
  • to_csv
  • to_excel
  • to_hdf
  • to_sql
  • to_json
  • to_html
  • to_stata
  • to_clipboard
• Fix modulo and integer division on Series,DataFrames to act similarly to float dtypes to returnnp.nan or np.inf as appropriate (GH 3590). This correct a numpy bug that treats integerand float dtypes differently.
  In [1]: p = pd.DataFrame({"first": [4, 5, 8], "second": [0, 0, 3]})
  In [2]: p % 0
  Out[2]:
  first second
  0 NaN NaN
  1 NaN NaN
  2 NaN NaN
  In [3]: p % p
  Out[3]:
  first second
  0 0.0 NaN
  1 0.0 NaN
  2 0.0 0.0
  In [4]: p / p
  Out[4]:
  first second
  0 1.0 NaN
  1 1.0 NaN
  2 1.0 1.0
  In [5]: p / 0
  Out[5]:
  first second
  0 inf NaN
  1 inf NaN
  2 inf inf
• Add squeeze keyword to groupby to allow reduction from DataFrame -> Series if groups are unique. This is a Regression from 0.10.1. We are reverting back to the prior behavior. This means groupby will return the same shaped objects whether the groups are unique or not. Revert this issue (GH 2893) with (GH 3596).
  In [2]: df2 = pd.DataFrame([{"val1": 1, "val2": 20},
  ...: {"val1": 1, "val2": 19},
  ...: {"val1": 1, "val2": 27},
  ...: {"val1": 1, "val2": 12}])
  In [3]: def func(dataf):
  ...: return dataf["val2"] - dataf["val2"].mean()
  ...:
  In [4]: # squeezing the result frame to a series (because we have unique groups)
  ...: df2.groupby("val1", squeeze=True).apply(func)
  Out[4]:
  0 0.5
  1 -0.5
  2 7.5
  3 -7.5
  Name: 1, dtype: float64
  In [5]: # no squeezing (the default, and behavior in 0.10.1)
  ...: df2.groupby("val1").apply(func)
  Out[5]:
  val2 0 1 2 3
  val1
  1 0.5 -0.5 7.5 -7.5
• Raise on iloc when boolean indexing with a label based indexer mask e.g. a boolean Series, even with integer labels, will raise. Since ilocis purely positional based, the labels on the Series are not alignable (GH 3631)
  This case is rarely used, and there are plenty of alternatives. This preserves theiloc API to be purely positional based.
  In [6]: df = pd.DataFrame(range(5), index=list("ABCDE"), columns=["a"])
  In [7]: mask = df.a % 2 == 0
  In [8]: mask
  Out[8]:
  A True

B False
C True
D False
E True
Name: a, dtype: bool

this is what you should use

In [9]: df.loc[mask]
Out[9]:
a
A 0
C 2
E 4

this will work as well

In [10]: df.iloc[mask.values]
Out[10]:
a
A 0
C 2
E 4
df.iloc[mask] will raise a ValueError

• The raise_on_error argument to plotting functions is removed. Instead, plotting functions raise a TypeError when the dtype of the object is object to remind you to avoid object arrays whenever possible and thus you should cast to an appropriate numeric dtype if you need to plot something.
• Add colormap keyword to DataFrame plotting methods. Accepts either a matplotlib colormap object (ie, matplotlib.cm.jet) or a string name of such an object (ie, ‘jet’). The colormap is sampled to select the color for each column. Please see Colormaps for more information. (GH 3860)
• DataFrame.interpolate() is now deprecated. Please useDataFrame.fillna() and DataFrame.replace() instead. (GH 3582,GH 3675, GH 3676)
• the method and axis arguments of DataFrame.replace() are deprecated
• DataFrame.replace ‘s infer_types parameter is removed and now performs conversion by default. (GH 3907)
• Add the keyword allow_duplicates to DataFrame.insert to allow a duplicate column to be inserted if True, default is False (same as prior to 0.12) (GH 3679)
• Implement __nonzero__ for NDFrame objects (GH 3691, GH 3696)
• IO api
  • added top-level function read_excel to replace the following, The original API is deprecated and will be removed in a future version
    from pandas.io.parsers import ExcelFile
    xls = ExcelFile("path_to_file.xls")
    xls.parse("Sheet1", index_col=None, na_values=["NA"])
    With
    import pandas as pd
    pd.read_excel("path_to_file.xls", "Sheet1", index_col=None, na_values=["NA"])
  • added top-level function read_sql that is equivalent to the following
    from pandas.io.sql import read_frame
    read_frame(...)
• DataFrame.to_html and DataFrame.to_latex now accept a path for their first argument (GH 3702)
• Do not allow astypes on datetime64[ns] except to object, andtimedelta64[ns] to object/int (GH 3425)
• The behavior of datetime64 dtypes has changed with respect to certain so-called reduction operations (GH 3726). The following operations now raise a TypeError when performed on a Series and return an empty Series when performed on a DataFrame similar to performing these operations on, for example, a DataFrame of slice objects:
  • sum, prod, mean, std, var, skew, kurt, corr, and cov
• read_html now defaults to None when reading, and falls back onbs4 + html5lib when lxml fails to parse. a list of parsers to try until success is also valid
• The internal pandas class hierarchy has changed (slightly). The previous PandasObject now is called PandasContainer and a newPandasObject has become the base class for PandasContainer as well as Index, Categorical, GroupBy, SparseList, andSparseArray (+ their base classes). Currently, PandasObjectprovides string methods (from StringMixin). (GH 4090, GH 4092)
• New StringMixin that, given a __unicode__ method, gets python 2 and python 3 compatible string methods (__str__, __bytes__, and__repr__). Plus string safety throughout. Now employed in many places throughout the pandas library. (GH 4090, GH 4092)

IO enhancements#

• pd.read_html() can now parse HTML strings, files or urls and return DataFrames, courtesy of @cpcloud. (GH 3477, GH 3605, GH 3606, GH 3616). It works with a single parser backend: BeautifulSoup4 + html5lib See the docs
  You can use pd.read_html() to read the output from DataFrame.to_html() like so
  In [11]: df = pd.DataFrame({"a": range(3), "b": list("abc")})
  In [12]: print(df)
  a b
  0 0 a
  1 1 b
  2 2 c
  In [13]: html = df.to_html()
  In [14]: alist = pd.read_html(html, index_col=0)
  In [15]: print(df == alist[0])
  a b
  0 True True
  1 True True
  2 True True
  Note that alist here is a Python list so pd.read_html() andDataFrame.to_html() are not inverses.
  • pd.read_html() no longer performs hard conversion of date strings (GH 3656).
• Added module for reading and writing Stata files: pandas.io.stata (GH 1512) accessible via read_stata top-level function for reading, and to_stata DataFrame method for writing, See the docs
• Added module for reading and writing json format files: pandas.io.jsonaccessible via read_json top-level function for reading, and to_json DataFrame method for writing, See the docsvarious issues (GH 1226, GH 3804, GH 3876, GH 3867, GH 1305)
• MultiIndex column support for reading and writing csv format files
  • The header option in read_csv now accepts a list of the rows from which to read the index.
  • The option, tupleize_cols can now be specified in both to_csv andread_csv, to provide compatibility for the pre 0.12 behavior of writing and reading MultIndex columns via a list of tuples. The default in 0.12 is to write lists of tuples and not interpret list of tuples as aMultiIndex column.
    Note: The default behavior in 0.12 remains unchanged from prior versions, but starting with 0.13, the default to write and read MultiIndex columns will be in the new format. (GH 3571, GH 1651, GH 3141)
  • If an index_col is not specified (e.g. you don’t have an index, or wrote it with df.to_csv(..., index=False), then any names on the columns index will be lost.
    In [16]: mi_idx = pd.MultiIndex.from_arrays([[1, 2, 3, 4], list("abcd")], names=list("ab"))
    In [17]: mi_col = pd.MultiIndex.from_arrays([[1, 2], list("ab")], names=list("cd"))
    In [18]: df = pd.DataFrame(np.ones((4, 2)), index=mi_idx, columns=mi_col)
    In [19]: df.to_csv("mi.csv")
    In [20]: print(open("mi.csv").read())
    c,,1,2
    d,,a,b
    a,b,,
    1,a,1.0,1.0
    2,b,1.0,1.0
    3,c,1.0,1.0
    4,d,1.0,1.0
    In [21]: pd.read_csv("mi.csv", header=[0, 1, 2, 3], index_col=[0, 1])
    Out[21]:
    c 1 2

  d a b
  a Unnamed: 2_level_2 Unnamed: 3_level_2
  1 1.0 1.0
  2 b 1.0 1.0
  3 c 1.0 1.0
  4 d 1.0 1.0

• Support for HDFStore (via PyTables 3.0.0) on Python3
• Iterator support via read_hdf that automatically opens and closes the store when iteration is finished. This is only for tables
  In [25]: path = 'store_iterator.h5'
  In [26]: pd.DataFrame(np.random.randn(10, 2)).to_hdf(path, 'df', table=True)
  In [27]: for df in pd.read_hdf(path, 'df', chunksize=3):
  ....: print(df)
  ....:
  0 1

0 0.713216 -0.778461
1 -0.661062 0.862877
2 0.344342 0.149565
0 1
3 -0.626968 -0.875772
4 -0.930687 -0.218983
5 0.949965 -0.442354
0 1
6 -0.402985 1.111358
7 -0.241527 -0.670477
8 0.049355 0.632633
0 1
9 -1.502767 -1.225492

• read_csv will now throw a more informative error message when a file contains no columns, e.g., all newline characters

Other enhancements#

• DataFrame.replace() now allows regular expressions on containedSeries with object dtype. See the examples section in the regular docsReplacing via String Expression
  For example you can do
  In [22]: df = pd.DataFrame({"a": list("ab.."), "b": [1, 2, 3, 4]})
  In [23]: df.replace(regex=r"\s.\s", value=np.nan)
  Out[23]:
  a b
  0 a 1
  1 b 2
  2 NaN 3
  3 NaN 4
  to replace all occurrences of the string '.' with zero or more instances of surrounding white space with NaN.
  Regular string replacement still works as expected. For example, you can do
  In [24]: df.replace(".", np.nan)
  Out[24]:
  a b
  0 a 1
  1 b 2
  2 NaN 3
  3 NaN 4
  to replace all occurrences of the string '.' with NaN.
• pd.melt() now accepts the optional parameters var_name and value_nameto specify custom column names of the returned DataFrame.
• pd.set_option() now allows N option, value pairs (GH 3667).
  Let’s say that we had an option 'a.b' and another option 'b.c'. We can set them at the same time:
  In [31]: pd.get_option('a.b')
  Out[31]: 2
  In [32]: pd.get_option('b.c')
  Out[32]: 3
  In [33]: pd.set_option('a.b', 1, 'b.c', 4)
  In [34]: pd.get_option('a.b')
  Out[34]: 1
  In [35]: pd.get_option('b.c')
  Out[35]: 4
• The filter method for group objects returns a subset of the original object. Suppose we want to take only elements that belong to groups with a group sum greater than 2.
  In [25]: sf = pd.Series([1, 1, 2, 3, 3, 3])
  In [26]: sf.groupby(sf).filter(lambda x: x.sum() > 2)
  Out[26]:
  3 3
  4 3
  5 3
  dtype: int64
  The argument of filter must a function that, applied to the group as a whole, returns True or False.
  Another useful operation is filtering out elements that belong to groups with only a couple members.
  In [27]: dff = pd.DataFrame({"A": np.arange(8), "B": list("aabbbbcc")})
  In [28]: dff.groupby("B").filter(lambda x: len(x) > 2)
  Out[28]:
  A B
  2 2 b
  3 3 b
  4 4 b
  5 5 b
  Alternatively, instead of dropping the offending groups, we can return a like-indexed objects where the groups that do not pass the filter are filled with NaNs.
  In [29]: dff.groupby("B").filter(lambda x: len(x) > 2, dropna=False)
  Out[29]:
  A B
  0 NaN NaN
  1 NaN NaN
  2 2.0 b
  3 3.0 b
  4 4.0 b
  5 5.0 b
  6 NaN NaN
  7 NaN NaN
• Series and DataFrame hist methods now take a figsize argument (GH 3834)
• DatetimeIndexes no longer try to convert mixed-integer indexes during join operations (GH 3877)
• Timestamp.min and Timestamp.max now represent valid Timestamp instances instead of the default datetime.min and datetime.max (respectively), thanks @SleepingPills
• read_html now raises when no tables are found and BeautifulSoup==4.2.0 is detected (GH 4214)

Experimental features#

• Added experimental CustomBusinessDay class to support DateOffsetswith custom holiday calendars and custom weekmasks. (GH 2301)
  Note
  This uses the numpy.busdaycalendar API introduced in Numpy 1.7 and therefore requires Numpy 1.7.0 or newer.
  In [30]: from pandas.tseries.offsets import CustomBusinessDay
  In [31]: from datetime import datetime

As an interesting example, let's look at Egypt where

a Friday-Saturday weekend is observed.

In [32]: weekmask_egypt = "Sun Mon Tue Wed Thu"

They also observe International Workers' Day so let's

add that for a couple of years

In [33]: holidays = ["2012-05-01", datetime(2013, 5, 1), np.datetime64("2014-05-01")]
In [34]: bday_egypt = CustomBusinessDay(holidays=holidays, weekmask=weekmask_egypt)
In [35]: dt = datetime(2013, 4, 30)
In [36]: print(dt + 2 * bday_egypt)
2013-05-05 00:00:00
In [37]: dts = pd.date_range(dt, periods=5, freq=bday_egypt)
In [38]: print(pd.Series(dts.weekday, dts).map(pd.Series("Mon Tue Wed Thu Fri Sat Sun".split())))
2013-04-30 Tue
2013-05-02 Thu
2013-05-05 Sun
2013-05-06 Mon
2013-05-07 Tue
Freq: C, dtype: object

Bug fixes#

• Plotting functions now raise a TypeError before trying to plot anything if the associated objects have a dtype of object (GH 1818,GH 3572, GH 3911, GH 3912), but they will try to convert object arrays to numeric arrays if possible so that you can still plot, for example, an object array with floats. This happens before any drawing takes place which eliminates any spurious plots from showing up.
• fillna methods now raise a TypeError if the value parameter is a list or tuple.
• Series.str now supports iteration (GH 3638). You can iterate over the individual elements of each string in the Series. Each iteration yields a Series with either a single character at each index of the originalSeries or NaN. For example,
  In [38]: strs = "go", "bow", "joe", "slow"
  In [32]: ds = pd.Series(strs)
  In [33]: for s in ds.str:
  ...: print(s)
  0 g
  1 b
  2 j
  3 s
  dtype: object
  0 o
  1 o
  2 o
  3 l
  dtype: object
  0 NaN
  1 w

2 e
3 o
dtype: object
0 NaN
1 NaN
2 NaN
3 w
dtype: object
In [41]: s
Out[41]:
0 NaN
1 NaN
2 NaN
3 w
dtype: object
In [42]: s.dropna().values.item() == "w"
Out[42]: True
The last element yielded by the iterator will be a Series containing the last element of the longest string in the Series with all other elements being NaN. Here since 'slow' is the longest string and there are no other strings with the same length 'w' is the only non-null string in the yielded Series.

• HDFStore
  • will retain index attributes (freq,tz,name) on recreation (GH 3499)
  • will warn with a AttributeConflictWarning if you are attempting to append an index with a different frequency than the existing, or attempting to append an index with a different name than the existing
  • support datelike columns with a timezone as data_columns (GH 2852)
• Non-unique index support clarified (GH 3468).
  • Fix assigning a new index to a duplicate index in a DataFrame would fail (GH 3468)
  • Fix construction of a DataFrame with a duplicate index
  • ref_locs support to allow duplicative indices across dtypes, allows iget support to always find the index (even across dtypes) (GH 2194)
  • applymap on a DataFrame with a non-unique index now works (removed warning) (GH 2786), and fix (GH 3230)
  • Fix to_csv to handle non-unique columns (GH 3495)
  • Duplicate indexes with getitem will return items in the correct order (GH 3455, GH 3457) and handle missing elements like unique indices (GH 3561)
  • Duplicate indexes with and empty DataFrame.from_records will return a correct frame (GH 3562)
  • Concat to produce a non-unique columns when duplicates are across dtypes is fixed (GH 3602)
  • Allow insert/delete to non-unique columns (GH 3679)
  • Non-unique indexing with a slice via loc and friends fixed (GH 3659)
  • Allow insert/delete to non-unique columns (GH 3679)
  • Extend reindex to correctly deal with non-unique indices (GH 3679)
  • DataFrame.itertuples() now works with frames with duplicate column names (GH 3873)
  • Bug in non-unique indexing via iloc (GH 4017); added takeable argument toreindex for location-based taking
  • Allow non-unique indexing in series via .ix/.loc and __getitem__ (GH 4246)
  • Fixed non-unique indexing memory allocation issue with .ix/.loc (GH 4280)
• DataFrame.from_records did not accept empty recarrays (GH 3682)
• read_html now correctly skips tests (GH 3741)
• Fixed a bug where DataFrame.replace with a compiled regular expression in the to_replace argument wasn’t working (GH 3907)
• Improved network test decorator to catch IOError (and thereforeURLError as well). Added with_connectivity_check decorator to allow explicitly checking a website as a proxy for seeing if there is network connectivity. Plus, new optional_args decorator factory for decorators. (GH 3910, GH 3914)
• Fixed testing issue where too many sockets where open thus leading to a connection reset issue (GH 3982, GH 3985, GH 4028,GH 4054)
• Fixed failing tests in test_yahoo, test_google where symbols were not retrieved but were being accessed (GH 3982, GH 3985,GH 4028, GH 4054)
• Series.hist will now take the figure from the current environment if one is not passed
• Fixed bug where a 1xN DataFrame would barf on a 1xN mask (GH 4071)
• Fixed running of tox under python3 where the pickle import was getting rewritten in an incompatible way (GH 4062, GH 4063)
• Fixed bug where sharex and sharey were not being passed to grouped_hist (GH 4089)
• Fixed bug in DataFrame.replace where a nested dict wasn’t being iterated over when regex=False (GH 4115)
• Fixed bug in the parsing of microseconds when using the formatargument in to_datetime (GH 4152)
• Fixed bug in PandasAutoDateLocator where invert_xaxis triggered incorrectly MilliSecondLocator (GH 3990)
• Fixed bug in plotting that wasn’t raising on invalid colormap for matplotlib 1.1.1 (GH 4215)
• Fixed the legend displaying in DataFrame.plot(kind='kde') (GH 4216)
• Fixed bug where Index slices weren’t carrying the name attribute (GH 4226)
• Fixed bug in initializing DatetimeIndex with an array of strings in a certain time zone (GH 4229)
• Fixed bug where html5lib wasn’t being properly skipped (GH 4265)
• Fixed bug where get_data_famafrench wasn’t using the correct file edges (GH 4281)

See the full release notes or issue tracker on GitHub for a complete list.

Contributors#

A total of 50 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

• Andy Hayden
• Chang She
• Christopher Whelan
• Damien Garaud
• Dan Allan
• Dan Birken
• Dieter Vandenbussche
• Dražen Lučanin
• Gábor Lipták +
• Jeff Mellen +
• Jeff Tratner +
• Jeffrey Tratner +
• Jonathan deWerd +
• Joris Van den Bossche +
• Juraj Niznan +
• Karmel Allison
• Kelsey Jordahl
• Kevin Stone +
• Kieran O’Mahony
• Kyle Meyer +
• Mike Kelly +
• PKEuS +
• Patrick O’Brien +
• Phillip Cloud
• Richard Höchenberger +
• Skipper Seabold
• SleepingPills +
• Tobias Brandt
• Tom Farnbauer +
• TomAugspurger +
• Trent Hauck +
• Wes McKinney
• Wouter Overmeire
• Yaroslav Halchenko
• conmai +
• danielballan +
• davidshinn +
• dieterv77
• duozhang +
• ejnens +
• gliptak +
• jniznan +
• jreback
• lexual
• nipunreddevil +
• ogiaquino +
• stonebig +
• tim smith +
• timmie
• y-p