A browser-based prototype of checkout-free retail vision. A pre-trained convolutional neural network (COCO-SSD / MobileNet v2) runs entirely on-device via TensorFlow.js — no video ever leaves this machine — detecting items in real time and maintaining a virtual cart as products enter and leave the frame.
🔒 Your camera feed is processed entirely on this device.
No video or images are ever uploaded.
The model runs in-browser via TensorFlow.js. Streaming raw video from hundreds of in-store cameras to the cloud is bandwidth-prohibitive. The PM decision is what to compute locally vs. escalate — the same signal-prioritization tradeoff at scale.
The confidence threshold is the operating point on the precision/recall curve. Too low floods the cart with false adds; too high misses real items. Where you set it is a business decision about customer trust and shrink — not just a technical one.
A real system swaps COCO's 80 generic classes for a retail-specific model, fuses multiple camera angles to handle occlusion, and escalates low-confidence events to a cloud reasoning layer (a Vision-Language Model) for the hard edge cases.
When you start the camera, each video frame is passed to COCO-SSD — a convolutional neural network with a MobileNet v2 backbone — running locally through TensorFlow.js. The model returns detected objects with class labels, bounding boxes, and confidence scores. The app draws those boxes on an overlaid canvas, then applies a small amount of product logic on top.
Two pieces of logic make it behave like a real store rather than a flickery demo. A debounce requires an item to appear for several consecutive frames before it's added — this suppresses momentary false detections. A presence timeout removes an item only after it has been gone from view for over a second — so brief occlusion (a hand passing over a product) doesn't drop it from the cart. Both are deliberate trust safeguards, not happy-path shortcuts.
The confidence slider adjusts the detection threshold, trading off precision and recall. Lowering it catches more items but admits false adds (higher recall, lower precision); raising it is safer but misses real items (lower recall, higher precision). In a real checkout-free store, the threshold is a business decision that balances false positives, missed detections, customer trust, and shrink.
Computer vision didn't start with deep learning. It's the product of six decades of work, and the breakthrough ideas trace back further than most people expect — to a pair of neuroscientists studying cats.
A CNN's core operation is the convolution: a small grid of learnable numbers (a filter) slides across the image pixel-by-pixel. At each position it multiplies itself against the pixel patch underneath and sums the result into a single number. One filter might learn to detect vertical edges, another horizontal ones — exactly like a Hubel-Wiesel orientation-selective neuron. The output is a "feature map" showing where that pattern appears in the image.
Stack these layers and the features grow in sophistication:
Between layers, pooling downsamples — keeping the strongest signal in each small region, so the network becomes robust to small shifts in position and scale. Because the same small filter is reused at every location across the whole image (weight sharing), CNNs use far fewer parameters than a fully-connected network would. That efficiency is why one runs live, in-browser, on a webcam feed without a GPU.
CNNs were inspired by biology but differ from it in important ways. Human vision is predictive — the brain constantly generates a model of the expected scene and checks incoming light against that prediction (which is why optical illusions fool us; they satisfy the prediction while violating the ground truth). It is selective — a tiny high-resolution fovea darts around a scene in rapid saccades rather than processing all pixels equally. And it carries a world model — you know an object still exists when it's hidden behind another (object permanence). A CNN, by contrast, is typically feedforward and does not maintain state across frames unless explicitly designed to do so. The frontier in 2026 is closing that gap by pairing fast CNN detection with transformer-based reasoning that begins to approach human-like context and understanding.
The winning architecture for a real checkout-free store is a hybrid: lightweight CNNs at the edge — on the camera itself — for fast, cheap, high-throughput detection; and a cloud reasoning layer (a VLM) that only sees the hard cases: obscured barcodes, new products, ambiguous hand-offs. That mirrors the edge-vs-cloud signal prioritization problem that appears at scale across any system with millions of sensing endpoints: you cannot move all the raw data, so you decide intelligently what to compute locally and what is worth the cost of escalating.
COCO-SSD is trained on the COCO dataset (Common Objects in Context) and recognises 80 object classes. Of those, this prototype maps a selected subset of COCO classes to SKUs, added to the virtual cart when confirmed — the subset of detected classes this prototype maps to products, grouped by category below. The remaining 23 (people, animals, vehicles, and road infrastructure) are still detected and drawn with bounding boxes, but carry no price.
23 classes detected but excluded — people, animals, vehicles, and road infrastructure.
The entire app lives in a single file: artifacts/smartcart-vision/index.html.
There is no build step, no bundler, and no npm dependencies. The page loads
TensorFlow.js and COCO-SSD from CDN, then all logic runs in the inline
<script> block at the bottom of the file. CSS is in a
single <style> block in <head>.
Supporting files (public/favicon.svg, public/manifest.json)
provide the app icon and PWA install metadata only.
Open index.html and find the SKU_MAP object. Add one line:
'coco class name': { price: X.XX, icon: '🆕', label: 'Display Name' },
The key must be an exact COCO-SSD class name (all lowercase, spaces allowed —
e.g. 'cell phone', 'sports ball'). Reload the page.
If the model detects that class, it will now appear in the cart with the price you set.
Items not in SKU_MAP are still detected and boxed (grey), but are not added to the cart.
Two constants in the DETECTION CONSTANTS section control how aggressively items are added and removed:
7) — number of consecutive video frames an item must appear in before it is added. At ~30 fps this is ~230 ms. Increase to reduce false adds; decrease for faster response.
1500 ms) — how long an item can be absent from frame before it is removed from the cart (Cart Mode only). Increase if items flicker out during occlusion; decrease for snappier removal.
confThreshold (default 60%). This is the precision/recall operating point — lower catches more items but admits false detections; higher is more conservative.
The app has two modes, toggled by the mode button. Internally they share the same
detectLoop() but diverge in how confirmed items are stored and displayed:
cart Map
(class → { conf, lastSeen }). Each frame, lastSeen is refreshed for
visible items. The presence-timeout pass removes any item whose lastSeen
is older than PRESENCE_TIMEOUT. The cart mirrors what is currently in frame.
scannerTotals
(class → { qty, conf }) and never removed by the timeout.
The cart Map is still used as a "currently visible" tracker so the same
physical item is not re-counted while it stays in frame — but once it disappears and
reappears, cart is clear of it and it can be scanned again (qty increments).
"Clear Cart" resets all three Maps (cart, pending, scannerTotals).
Cart Mode state:
pending[cls]++ each frame → if count ≥ ADD_FRAMES → cart.set(cls, {...})
presence pass each frame → if now - lastSeen > TIMEOUT → cart.delete(cls)
Scanner Mode state:
same ADD_FRAMES gate → scannerTotals[cls].qty++ (permanent)
presence pass → cart.delete(cls) only (so it can be re-scanned)
renderCart reads scannerTotals, not cart
The no-build-step design is intentional for a demo: it makes the prototype trivially shareable (one file, any static host, opens from disk), removes all toolchain friction, and keeps the entire implementation readable in a single scroll. The tradeoff is that scaling to a real product would require splitting concerns — separate JS modules, a proper model-serving layer, and a backend cart/session API. Those are the right next steps for a production system, but wrong for a prototype where speed of demonstration is the priority.
Effective date: June 2026 · Last updated: June 2026
SmartCart Checkout collects a small amount of anonymous, non-personal session data when you use the app. Specifically:
sessionStorage. It is cleared automatically when you close the tab or browser window. It is not linked to your identity in any way.session_start, camera_started, item_added, scanner_mode_toggled, session_end.We do not collect video, images, or frames from your camera. All object detection runs entirely on your device via TensorFlow.js. No visual data ever leaves your machine. We do not collect your name, email, IP address, or any personally identifying information. No cookies are set — ever.
Session events are stored in a Replit-managed PostgreSQL database. The database is not shared with or sold to any third party. No advertising or analytics platforms (Google Analytics, Mixpanel, etc.) are used.
Session records are retained for 90 days and then permanently deleted.
Only the app owner can query the database directly, using Replit's built-in database tooling. No one else has access.
No third-party AI services are used. Object detection is performed by the COCO-SSD / MobileNet v2 model, which runs entirely in your browser via TensorFlow.js. The model weights are fetched once from a public CDN (cdnjs.cloudflare.com / cdn.jsdelivr.net) and cached locally by your browser.
Because we store only anonymous session IDs and no personal data, there is nothing that can be traced back to you.
This app sets no cookies of any kind. The session ID lives only in sessionStorage — a browser
storage mechanism that is scoped to the current tab and erased when you close it. No consent banner is needed.
Camera stays on your device. No personal data collected. No cookies. Anonymous session events stored for 90 days. Not shared with anyone.
Effective date: June 2026
SmartCart Checkout is a prototype demonstration of checkout-free retail computer vision. By using this app you agree to the following:
This app is provided "as is" without warranties of any kind, express or implied. Detection accuracy depends on lighting, camera quality, and the limitations of the COCO-SSD model. The app is a prototype and is not suitable for production retail use without further engineering, validation, and compliance review.
The owner reserves the right to modify, suspend, or permanently shut down this service at any time and without notice. No guarantee of uptime or continued availability is made.
To the maximum extent permitted by applicable law, the owner is not liable for any damages arising from your use of or inability to use this service.