Pandas DataFrame duplicated() Method – Be on the Right Side of Change (original) (raw)
Preparation
Before any data manipulation can occur, two (2) new libraries will require installation.
- The Pandas library enables access to/from a DataFrame.
- The NumPy library supports multi-dimensional arrays and matrices in addition to a collection of mathematical functions.
To install these libraries, navigate to an IDE terminal. At the command prompt ($), execute the code below. For the terminal used in this example, the command prompt is a dollar sign ($). Your terminal prompt may be different.
$ pip install pandas
Hit the <Enter> key on the keyboard to start the installation process.
$ pip install numpy
Hit the <Enter> key on the keyboard to start the installation process.
If the installations were successful, a message displays in the terminal indicating the same.
Feel free to view the PyCharm installation guide for the required libraries.
Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.
import pandas as pd import numpy as np
The duplicated() method returns a Series of boolean values indicating duplicate row(s).
The syntax for this method is as follows:
DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)
| Parameter | Description |
|---|---|
| subset | Specify the column(s) to locate the duplicates. By default, all columns. |
| keep | Determines what duplicates to keep, first or the last occurrence. By default, first. |
| inplace | If False creates a copy of the DataFrame/Series. By default, False. If True, the original DataFrame/Series updates. |
| ignore_index | If True, the returning axis will start the numbers from 0 – n value. By default, False. |
For this example, a DataFrame is created that contains four (4) rows.
Notice that GMC has two (2) identical records in the DataFrame.
Code Example 1
df = pd.DataFrame({'Make': ['Honda', 'GMC', 'GMC', 'Ford'], 'Model': ['Civic', 'Canyon', 'Canyon', 'Mustang'], 'Rating': [4, 3.5, 3.5, 15]})
result = df.drop_duplicates() print(result)
- Line [1] creates a DataFrame from a Dictionary of Lists and saves it to
df. - Line [2] removes the duplicate row(s).
- Line [3] outputs the result to the terminal.
Output:
| | Make | Model | Ratings | | | ------ | ----- | ------- | ---- | | 0 | Honda | Civic | 4.0 | | 1 | GMC | Canyon | 3.5 | | 3 | Ford | Mustang | 15.0 |
Another way to remove duplicate rows is to use the NumPy library.
Code – Example 2
df = np.array([[1,8,3,3,4], [1,8,9,9,4], [1,8,3,3,4]])
new_array = [tuple(row) for row in df]
result = np.unique(new_array, axis=0)
print(result)
- Line [1] creates a DataFrame from a List of Lists and saves it to
df. - Line [2] removes the duplicate row(s).
- Line [3] outputs the result to the terminal.
Output
[[1 8 3 3 4] [1 8 9 9 4]]
More Pandas DataFrame Methods
Feel free to learn more about the previous and next pandas DataFrame methods (alphabetically) here:
Also, check out the full cheat sheet overview of all Pandas DataFrame methods.