Data Processing
How to clean, filter, and prepare e-scooter data.
Overview
Raw e-scooter telemetry requires extensive processing before it can be used for wheelchair accessibility mapping. The data is filtered for noise, merged across multiple ride files, and transformed into clean arrays suitable for surface quality assessment.
Speed <0.5 km/h
Validate GPS:
Accuracy <10 m, no gaps >5 seconds
Sensor checks:
Remove saturation, ensure 30 Hz consistency
z_std:
Rolling standard deviation of Z-axis (30-sample window)
speed_kmh:
Convert raw speed ÷ 10 → km/h (unit: 1/10 km/h)
coordinates:
GPS lat/lon pairs
quality_score:
Combined GPS + sensor quality metric
Z-score >3 standard deviations
Smoothing:
Butterworth filter, 4th order, 10 Hz cutoff
Gap handling:
Linear interpolation for small gaps
Quality Metrics
Filtering inevitably discards data, so it has to be worth the cost. Three measurements track what the cleaning stage keeps, how much clearer the vibration signal becomes, and how long a typical ride takes to process on ordinary hardware.
Rolling Window Smoothing
Rolling window analysis reduces noise by averaging z_std values over a 30-sample window. This comparison shows the original z_std values alongside the smoothed rolling window values, demonstrating how noise spikes are filtered while preserving true surface roughness patterns.
Filter Comparison
Different filtering methods can be applied to z_std values to reduce sensor noise. The moving average filter uses a centered window, while exponential smoothing applies weighted averaging that preserves recent trends. Both methods effectively remove high-frequency mechanical vibrations while maintaining surface roughness signals.