Where it began
An ingredient list is dense, printed small, and full of codes like INS 471. Most people never read one. I wanted to know whether an app could read it for you and explain it plainly, with every judgement traceable back to a source.
It turned out to be a good place to work on the problem from both ends: reading messy text off a real wrapper, and working out what a fair, explainable grade should even be.
How it works
- Find the product. Scan a barcode, and the app looks for the product in Open Food Facts.
- Or read the label. Photograph the ingredient list. Text recognition runs on the phone itself. In the development build, a hosted vision model reads the whole list, and OCR then checks every word it returned, so the second reader keeps the first one honest.
- Understand the list. Each ingredient is parsed and matched to a known ingredient or additive from a pinned reference list.
- Grade it. The result is a single letter, based on how processed the product is and where its ingredients sit in the list, with reasons that cite where they come from.
- Keep it private. Saved results stay on the phone, in an encrypted database. Label photos are temporary.
Decisions along the way
I keep a written record of each design decision and why I made it. In the first month the direction changed often, and sometimes a lot.
-
India first
Open Food Facts is strongest in Europe. For India, in 2024, it had about 10,000 products, and only around 2% of them carried a Nutri-Score. I chose India anyway, which meant the app would have no existing grades to lean on.
-
Ingredients before nutrition
Among the 79 most-scanned Indian products, Nutri-Score could grade exactly one, while 67 had an ingredient list. The numbers pointed one way, so the headline grade moved to the ingredients, using the NOVA classification of how processed a food is. Nutrition was left for later.
-
A vision model reads, OCR checks
On curved wrappers, OCR broke additive codes, and fixed rules could not tell where the list ended. In a trial, a vision model read the whole list correctly, but it also filled in words hidden by the packet’s seam. So now the model reads, OCR checks every word, and anything the model added gets flagged. A good guess is still a guess.
-
Two axes, one letter
Graded on processing alone, most products bunched together at D or below, and a grade that says the same thing about nearly everything doesn’t tell you much. The grade now also weighs where ingredients appear in the list, and aims to give one clear letter to every product with a readable list.
-
Measure instead of patch
I had been fixing unrecognised ingredients one name at a time, and there was always another one. Measured on 546 real Indian ingredient lists, about two thirds of ingredient mentions were recognised. Now I’m building recognition that is measured against a held-out set, instead of pieced together from one-off fixes.
Where it stands
For now it runs as an Android development build. The grading method is still changing, nutrition is paused, and a release waits on a privacy review. One honest note: the numbers above come from my own measurements on samples of Open Food Facts data, not from tests in shops. They describe the database. Real shop shelves are still untested.
What I learned
A grade that works on one product in seventy-nine isn’t really a grade.
Processing isn’t health
The first ingredient grade read processing as health. On a sample of 399 popular Indian products, 189 of the 211 that got a letter landed on D or D–E, and sugar, salt and refined palm oil on their own each got a B, while a plain list of whole wheat flour, water, yeast, salt and soya lecithin got a D. In India, ingredients have to be printed from most to least, by weight or volume, so where an ingredient sits in the list is a fact I can check. Turning NOVA groups into letters is my own choice, not an official scale, and the app should say so.
Two readers
In one trial, the vision model turned “…HITE” in an allergen line into WHITE, where the word on the pack may have been SULPHITE. So a word counts only if OCR saw it too, and anything else is marked uncertain. If any word in the allergen line is uncertain, the app says it can’t fully check the pack. It never says an allergen is absent.
Counting instead of patching
The unrecognised names were a long tail: in the development set, 858 different names, and 755 of them appeared only once. Adding names by hand would never catch up. Now the app reads a name in parts, like “mango pulp” as mango plus a form, and every change is measured on the held-out set. A change may not make those numbers worse, and it may never make an allergy result less cautious. That rule earned its place when a review found early clean-up rules quietly dropping text the allergy checks needed.
I stopped fixing the last scan and started asking where I actually stand.
Credits
- Product data from Open Food Facts, under the Open Database License.
- The NOVA food classification, by Monteiro and colleagues.
The code is private while the app is in development.