Case Studies » Ameru » Mlops

Closing the Loop with MLOps and User Feedback After Deployment

Field feedback on a regular cadence feeds the MLOps pipeline, improving sorting models while keeping users engaged in the learning process.

Applied Principles:

Architect for Flow Accelerate

Observed domains and modes:

System Stewardship Structure Commit Evolve Delivery Operate Monitor

Improving Sorting Accuracy After Deployment with User Feedback and MLOps

Context

Once deployed, Ameru’s bins operate under highly variable real-world conditions; Lighting changes, waste composition differs, user behavior varies, etc. Sorting accuracy therefore cannot be fully validated before deployment.

From the beginning, Ameru assumed that learning must continue after installation. The system had to generate signals from real use and feed them back into the development system. Ameru also learned that users perceive this learning loop positively, feeling acknowledged. The interface therefore invites feedback so users see that the system reacts to their input.

Architect for Flow

Structure The architecture separates deployed hardware from evolving AI models. The bin remains stable while models evolve through controlled training cycles.

Operate The user interface provides a simple feedback mechanism. After classification, the screen invites the user to confirm or reject the result using a thumbs up or thumbs down. The interaction takes one optional tap and does not interrupt the disposal process.

Monitor Negative feedback generates candidate signals for investigation. Images associated with rejected classifications are stored and reviewed manually. Positive feedback is currently ignored to avoid reinforcing accidental confirmations.

Structure Privacy requirements influenced architectural decisions early and led to a carefully crafted privacy policy. The camera points directly into the receptacle and cannot capture the surrounding environment. Images contain only the disposed item. The system stores no personal identifiers.

Accelerate

Evolve Ameru established an MLOps pipeline from the beginning. Early training relied on internally generated datasets. These experiments exposed limits of image-based classification and led to a shift toward item-based recognition.

Monitor User feedback later provided additional real-world examples. Engineers review flagged images on a monthly cadence. Confirmed misclassifications are added to the curated training dataset. Evolve Retraining incorporates these verified examples. Model quality is checked against key performance metrics to prevent regression.

Commit Improved models are deployed through over-the-air updates. Hardware installations remain unchanged while classification performance evolves.

Outcome

Sorting accuracy improves after deployment instead of degrading over time. Real-world variability becomes a source of training data. Manual review protects model quality while user feedback exposes edge cases. Ameru can expand deployments while continuously improving classification performance without modifying installed hardware.

External Resources

These videos show the evolution of the Ameru Smart Bin: (1) early prototype, (2) dealing with exceptions, (3) device sorting performance.

Early prototpye
Dealing with exceptions
Sorting performance

The Principles

More details on the principles

  • Define & Align (Value Thinking)
  • Structure & Scale (Architect for Flow)
  • Build & Validate (Shift Left)
  • Operate & Evolve (Accelerate)

The Velocity Loop

More details on the Velocity Loop