The Problem: Real-Time Pose Detection on Mid-Range Hardware

FitLife Inc. came to us with a clear goal: build an AI fitness app that counts exercise repetitions automatically using the phone's camera, provides personalised coaching, and runs on Android devices costing $150 or less.

The challenge is that real-time pose detection is computationally expensive. Running a full pose estimation model at 30fps on a flagship phone is feasible. Running it reliably on a mid-range Android device, the Redmi Note, the Samsung A-series, requires serious optimisation.

This is the story of how we achieved 91% rep-count accuracy with sub-50ms landmark detection on budget hardware.

Choosing the Right Pose Estimation Model

We evaluated three approaches:

• MediaPipe Pose (BlazePose), 33 keypoints, optimised for mobile, runs on CPU
• MoveNet Lightning, 17 keypoints, extremely fast, slightly less accurate
• PoseNet (TensorFlow.js), web-focused, not optimal for native mobile

We chose MediaPipe Pose via the google_ml_kit Flutter plugin for its accuracy at the shoulder, elbow, wrist, hip, knee, and ankle keypoints, the six joints needed for most gym exercises.

Rep Counting Algorithm

The naive approach, counting times a joint crosses a threshold angle, fails in the real world. Users don't perform exercises identically. Range of motion varies. Camera angle changes.

Our algorithm uses a finite state machine with hysteresis:

• State: UP, joint angle > upper threshold (e.g., 160° for a bicep curl)
• State: DOWN, joint angle < lower threshold (e.g., 60° for a bicep curl)
• A rep is counted on each UP→DOWN→UP transition

The thresholds are personalised per user on first use using a calibration set of 3 reps. This alone improved accuracy from ~72% to ~88%.

The final 3% came from smoothing the joint angle signal with a 5-frame moving average to filter camera shake and detection jitter.

TensorFlow Lite Model Optimisation

The MediaPipe model runs at full precision (float32) by default. On mid-range Android, this consumes ~45% CPU, leaving nothing for the UI thread.

We applied post-training quantisation to INT8:

• Model size: 6.9MB → 1.8MB
• Inference time: 38ms → 14ms per frame
• Accuracy drop: < 1.5% on our test set

We also pinned inference to the device's DSP (Digital Signal Processor) using TFLite's NNAPI delegate, which offloads computation from the CPU entirely on supported chipsets.

The Flutter Architecture

The app follows a clear separation:

• Camera layer, camera Flutter plugin, set to 720p 30fps
• Inference layer, runs on a separate Dart isolate to never block the UI
• Pose overlay, CustomPainter draws skeleton lines over the camera feed at 30fps
• Rep counter, state machine consuming pose data from the inference layer
• Coaching engine, FastAPI backend that receives session data and returns personalised program adjustments

The isolate architecture was critical. Without it, inference blocks the UI thread and the camera feed stutters, destroying the user experience.

Results

After 6 weeks of development and 3 weeks of on-device testing across 12 different Android devices:

• 91% rep-count accuracy across squats, deadlifts, bench press, bicep curls, and pull-ups
• Sub-50ms landmark detection on all tested devices including Redmi Note 11
• 40% improvement in 30-day user retention compared to the client's previous manual logging app
• Launched in 3 months from first commit to Play Store submission

The key insight: hardware-aware optimisation is not optional for AI mobile apps. The difference between a great experience and an unusable one is often a single TFLite quantisation step.