The Challenge: ML in the Browser at Scale

Browser-based AR is technically remarkable and commercially fragile. You're running a 468-keypoint face mesh model inside a JavaScript runtime, on hardware you don't control, across browsers with varying WebGL support levels.

For GlamTech Ltd., we shipped this to millions of users across 40+ countries on devices ranging from a 2024 iPhone Pro to a 2019 mid-range Android with 3GB RAM.

Here's what actually broke in production, and how we fixed it.

TensorFlow.js Face Mesh: The Baseline

TF.js MediaPipe Face Mesh detects 468 facial landmarks at 15, 30fps depending on hardware. The model runs via WebGL backend for GPU acceleration, or WASM backend as CPU fallback.

Our baseline implementation was straightforward: load the model, run detection on each video frame, project makeup overlays using Three.js.

On a MacBook Pro, this ran at 28fps. On a Samsung Galaxy A32, it ran at 6fps. That's unusable for a makeup try-on experience.

Optimisation 1: Adaptive Frame Skipping

We don't need to run detection on every frame. The face doesn't teleport between frames. Our adaptive frame skip runs detection every N frames based on measured device performance:

• < 2s inference time: run every frame
• 2, 4s: run every 2 frames, interpolate landmarks between
• > 4s: run every 3 frames, use velocity-based landmark prediction

This brought the A32 from 6fps to 18fps, above the perceptual smooth threshold.

Optimisation 2: Custom Colour Calibration Pipeline

The most common complaint in user testing was that makeup colours looked wrong. A lipstick showing as bright red on the product page rendered as dark maroon in the try-on.

The cause: the video feed colour space is sRGB, Three.js renders in linear colour space, and the composited output is in sRGB again. Each conversion introduced colour shift.

We built a custom calibration pipeline: sample 10 pixels from the video feed, compare to the known-good product colour, compute an LUT (lookup table) correction. Applied per-user per-session, this eliminated perceptible colour mismatch.

Results

• 23% improvement in tracking accuracy via custom model fine-tuning on diverse skin tones
• 40% sales uplift in the first month post-launch vs. standard product images
• 95% positive user feedback in post-session surveys
• Smooth performance (>18fps) on 94% of devices in our target market