Pandas Best Practices

Code Style and Structure

• Write concise, technical responses with accurate Python examples
• Prioritize reproducibility in data analysis workflows
• Use functional programming; avoid unnecessary classes
• Prefer vectorized operations over explicit loops
• Use descriptive variable names reflecting data content
• Follow PEP 8 style guidelines

DataFrame Creation and I/O

• Use

  pd.read_csv()

  ,

  pd.read_excel()

  ,

  pd.read_json()

  with appropriate parameters
• Specify

  dtype

  parameter to ensure correct data types on load
• Use

  parse_dates

  for automatic datetime parsing
• Set

  index_col

  when the data has a natural index column
• Use

  chunksize

  for reading large files incrementally

Data Selection

• Use

  .loc[]

  for label-based indexing
• Use

  .iloc[]

  for integer position-based indexing
• Avoid chained indexing (e.g.,

  df['col'][0]

  ) - use

  .loc

  or

  .iloc

  instead
• Use boolean indexing for conditional selection:

  df[df['col'] > value]

• Use

  .query()

  method for complex filtering conditions

Method Chaining

• Prefer method chaining for data transformations when possible
• Use

  .pipe()

  for applying custom functions in a chain
• Chain operations like

  .assign()

  ,

  .query()

  ,

  .groupby()

  ,

  .agg()

• Keep chains readable by breaking across multiple lines

Data Cleaning and Validation

Missing Data

• Check for missing data with

  .isna()

  and

  .info()

• Handle missing data appropriately:

  .fillna()

  ,

  .dropna()

  , or imputation
• Use

  pd.NA

  for nullable integer and boolean types
• Document decisions about missing data handling

Data Quality Checks

• Implement data quality checks at the beginning of analysis
• Validate data types with

  .dtypes

  and convert as needed
• Check for duplicates with

  .duplicated()

  and handle appropriately
• Use

  .describe()

  for quick statistical overview

Type Conversion

• Use

  .astype()

  for explicit type conversion
• Use

  pd.to_datetime()

  for date parsing
• Use

  pd.to_numeric()

  with

  errors='coerce'

  for safe numeric conversion
• Utilize categorical data types for low-cardinality string columns

Grouping and Aggregation

GroupBy Operations

• Use

  .groupby()

  for efficient aggregation operations
• Specify aggregation functions with

  .agg()

  for multiple operations
• Use named aggregation for clearer output column names
• Consider

  .transform()

  for broadcasting results back to original shape

Pivot Tables and Reshaping

• Use

  .pivot_table()

  for multi-dimensional aggregation
• Use

  .melt()

  to convert wide to long format
• Use

  .pivot()

  to convert long to wide format
• Use

  .stack()

  and

  .unstack()

  for hierarchical index manipulation

Performance Optimization

Memory Efficiency

• Use categorical data types for low-cardinality strings
• Downcast numeric types when appropriate
• Use

  pd.eval()

  and

  .eval()

  for large expression evaluation

Computation Speed

• Use vectorized operations instead of

  .apply()

  with row-wise functions
• Prefer built-in aggregation functions over custom ones
• Use

  .values

  or

  .to_numpy()

  for NumPy operations when faster

Avoiding Common Pitfalls

• Avoid iterating with

  .iterrows()

  - use vectorized operations
• Don't modify DataFrames while iterating
• Be aware of SettingWithCopyWarning - use

  .copy()

  when needed
• Avoid growing DataFrames row by row - collect in list and create once

Time Series Operations

• Use

  DatetimeIndex

  for time series data
• Leverage

  .resample()

  for time-based aggregation
• Use

  .shift()

  and

  .diff()

  for lag operations
• Use

  .rolling()

  and

  .expanding()

  for window calculations

Merging and Joining

• Use

  .merge()

  for SQL-style joins
• Specify

  how

  parameter: 'inner', 'outer', 'left', 'right'
• Use

  validate

  parameter to check join cardinality
• Use

  .concat()

  for stacking DataFrames

Key Conventions

• Import as

  import pandas as pd

• Use

  snake_case

  for column names when possible
• Document data sources and transformations
• Keep notebooks reproducible with clear cell execution order