Kenhuang Academy
  • English

      • Introduction to Pandas

      What is Pandas?

      Pandas is one of the most important data analysis libraries in Python, named after "Panel Data". It provides high-performance, easy-to-use data structures and data analysis tools, and is a core tool in the data science and analytics field.

      🎯 Core Features of Pandas

      1. Powerful Data Structures

      • Series: One-dimensional labeled array, similar to a labeled list
      • DataFrame: Two-dimensional labeled data structure, similar to an Excel spreadsheet or SQL table

      2. Flexible Data Processing

      • Data reading and writing (CSV, Excel, JSON, SQL, etc.)
      • Data cleaning and preprocessing
      • Data transformation and reshaping
      • Missing data handling

      3. Efficient Data Analysis

      • Descriptive statistics
      • Data grouping and aggregation
      • Data merging and joining
      • Time series analysis

      🚀 Why Choose Pandas?

      Advantages Comparison

      FeaturePandasExcelSQL
      Data Volume HandlingLarge datasetsSmall to mediumLarge datasets
      Programming FlexibilityVery highLimitedMedium
      Data VisualizationIntegrated matplotlibBuilt-in chartsRequires other tools
      Automation LevelFully automatedManual operationsScriptable
      Learning CurveMediumSimpleMedium

      📊 Pandas Position in Data Science

      # Typical data science workflow
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# 1. Data acquisition
data = pd.read_csv('data.csv')

# 2. Data exploration
print(data.info())
print(data.describe())

# 3. Data cleaning
data_clean = data.dropna()

# 4. Data analysis
result = data_clean.groupby('category').mean()

# 5. Data visualization
result.plot(kind='bar')
plt.show()

      🏢 Pandas Use Cases

      1. Business Analytics

      # Sales data analysis example
sales_data = pd.DataFrame({
    'Date': ['2024-01-01', '2024-01-02', '2024-01-03'],
    'Product': ['A', 'B', 'A'],
    'Quantity': [100, 150, 120],
    'Revenue': [1000, 2250, 1200]
})

# Calculate total sales by product
product_sales = sales_data.groupby('Product')['Quantity'].sum()
print(product_sales)

      2. Financial Analysis

      # Stock data analysis example
stock_data = pd.DataFrame({
    'Date': pd.date_range('2024-01-01', periods=5),
    'Open': [100, 102, 101, 103, 105],
    'Close': [102, 101, 103, 105, 107],
    'Volume': [1000, 1200, 800, 1500, 1100]
})

# Calculate daily return
stock_data['Return'] = stock_data['Close'].pct_change()
print(stock_data)

      3. Scientific Research

      # Experiment data analysis example
experiment_data = pd.DataFrame({
    'Group': ['A', 'A', 'B', 'B', 'C', 'C'],
    'Measurement': [23.5, 24.1, 26.8, 27.2, 22.1, 21.9],
    'Temperature': [20, 20, 25, 25, 15, 15]
})

# Calculate statistics by group
stats = experiment_data.groupby('Group')['Measurement'].agg(['mean', 'std'])
print(stats)

      🔧 Pandas Ecosystem

      Pandas integrates closely with other Python libraries:

      Data Processing Pipeline

      # Complete data processing workflow
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

# Data reading (Pandas)
df = pd.read_csv('dataset.csv')

# Data preprocessing (Pandas + NumPy)
df_clean = df.fillna(df.mean())
X = df_clean[['feature1', 'feature2']]
y = df_clean['target']

# Data visualization (Pandas + Matplotlib/Seaborn)
df_clean.hist(figsize=(12, 8))
plt.show()

# Machine learning (Scikit-learn)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = LinearRegression()
model.fit(X_train, y_train)

      📈 Pandas Development History

      • 2008: Wes McKinney started development at AQR Capital Management
      • 2009: First public release
      • 2012: Became a NumFOCUS project
      • 2017: Released version 1.0
      • 2020: Released stable version 1.0
      • Present: Continuous active development with rich community contributions

      🌟 Core Concepts of Pandas

      1. Data Alignment

      # Automatic data alignment example
s1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])
s2 = pd.Series([4, 5, 6], index=['b', 'c', 'd'])

# Automatic index alignment
result = s1 + s2
print(result)
# a    NaN
# b    6.0
# c    8.0
# d    NaN

      2. Missing Data Handling

      # Smart missing data handling
data = pd.DataFrame({
    'A': [1, 2, None, 4],
    'B': [5, None, 7, 8],
    'C': [9, 10, 11, None]
})

# Multiple handling methods
print("Drop missing values:")
print(data.dropna())

print("\nFill missing values:")
print(data.fillna(0))

print("\nForward fill:")
print(data.fillna(method='ffill'))

      3. Flexible Indexing

      # Multi-level index example
index = pd.MultiIndex.from_tuples([
    ('A', 1), ('A', 2), ('B', 1), ('B', 2)
], names=['Letter', 'Number'])

df = pd.DataFrame({
    'Value': [10, 20, 30, 40]
}, index=index)

print(df)
print("\nSelect group A data:")
print(df.loc['A'])

      🎓 Suggested Learning Path

      Beginner Path

      1. Basic Concepts: Understand Series and DataFrame
      2. Data I/O: Master handling common data formats
      3. Basic Operations: Indexing, selection, filtering
      4. Data Cleaning: Handle missing values, duplicates
      5. Simple Analysis: Descriptive statistics, basic aggregation

      Intermediate Path

      1. Advanced Indexing: Multi-level indexing, time-based indexing
      2. Data Reshaping: pivot, melt, stack/unstack
      3. Data Merging: merge, join, concat
      4. Time Series: Date handling, resampling
      5. Performance Optimization: Vectorized operations, memory management

      Expert Path

      1. Custom Functions: apply, transform, agg
      2. Extension Features: Plugin development, custom accessors
      3. Big Data Processing: Chunked processing, Dask integration
      4. Performance Tuning: Cython, Numba acceleration
      5. Production Deployment: Data pipelines, automated workflows

      💡 Best Practices Preview

      1. Code Style

      # Recommended chained operations
result = (df
    .query('age > 18')
    .groupby('category')
    .agg({'sales': 'sum', 'profit': 'mean'})
    .sort_values('sales', ascending=False)
    .head(10)
)

      2. Performance Considerations

      # Use vectorized operations instead of loops
# Not recommended
for i in range(len(df)):
    df.loc[i, 'new_col'] = df.loc[i, 'col1'] * df.loc[i, 'col2']

# Recommended
df['new_col'] = df['col1'] * df['col2']

      3. Memory Management

      # Optimize data types
df['category'] = df['category'].astype('category')
df['small_int'] = df['small_int'].astype('int8')

      🔗 Related Resources

      📝 Chapter Summary

      Pandas is the core tool for Python data analysis with the following characteristics:

      ✅ Powerful Data Structures: Series and DataFrame
      ✅ Rich Functionality: Data I/O, cleaning, analysis, visualization
      ✅ Excellent Ecosystem: Seamless integration with NumPy, Matplotlib, Scikit-learn, etc.
      ✅ Active Community: Continuous development, comprehensive documentation
      ✅ Wide Applications: Business, finance, research, and various other fields

      In the next chapter, we will learn how to install and configure the Pandas development environment.

      Next Chapter: Pandas Installation