Speed up proof & price addition

[raphodn]

Currently

@serpico (extracted from #523)

In the same spirit/related, I was adding prices yesterday based on price tag's pictures I took the same day, for reference I timed myself doing it, for 16 prices it took me 54'( with a some back and forth between OFF and Open Price webapp to create products which didn't exist, but I believe that's not the bulk of the time spent ) so it took 3'40" per price, although I'm willing to admit my workflow could be enhanced, I do use a text file from which I copy/paste the EAN for the products I track, saving some time I think.
I noticed that for a session like that, the type of proof, the location, the date and currency would be the same and the user would save time if those were once entered at the beginning of the so-called "session" and "locked", some kind of streamlined interface without all those field after initially being entered, ideally making use of the keyboard's Tab key to jump from one field to the next...maybe that's mostly what we have for the receipt mode. An additional "lockable" parameter would be EAN as opposed to category - ideally jumping straight the the EAN input box to paste the relevant EAN.
After checking what the receipt mode UI looks like it would be a variant of it, if it's deemed worth exploring I'll expand in a specific issue and share screenshot of my vision of the UI, based on the current receipt mode.
Now the question is how many user engage ( percentage of prices coming from this opposed to receipt ) in this, how much time do we save ( here's I'm thinking a stopwatch feature could be useful although it might scare away user when they realize how much time they spend for so few prices - so maybe make it bonus interface ) and is it worth the time coding this specific interface.

Related issues

• Repurpose single price addition workflow to speed up single price / single picture case open-prices-frontend#961
• Multiple proof upload page open-prices-frontend#983
• Proof: add new "consumption-related" fields on receipts: total amount, items count #387 (comment)
• Leverage Hunger Games to crowdsource price addition (from price tag images) #523

[raphodn]

I would definitely be curious of a screenshot (or quick video ?) of your setup !

I presume you have 1 proof image per price in your case ? So it does require a bit of time to first upload the proof, and then add the single price. The current price addition workflow is indeed better for multiple prices.

This discussion would help in offloading the time spent per proof/price and leave it to the community. 3'40 per price is too long 😢

And we could use some image detection/cropping

• split images with multiple price tags, into images with only 1 single price tag, to make it more distributable on Hunger
• having OCR - for instance Proof: send to Google Cloud Vision, and store the matching JSON output #320 - would pre-fill the price & barcode forms, and have the users only "double check"

[raphodn]

fyi my current workflow :

• i take pictures with my phone (stored in the gallery), and add the proofs in the OP web app via the phone. my pictures always have at least 2 or 3 price tags on them
• once home, I open my laptop, connect to my dashboard, and for each proof add the related prices
  • my laptop screen is small, so sometimes by zooming on my phone image (phone next to my laptop)
  • sometimes opening the proof image in full size and displaying it side by side (but the web app compresses the images, the quality is not always enough...)

[raphodn]

@serpico

• I take pictures with my phone (stored in the gallery), for specific reasons I would rarely have 2 or 3 price tags in one picture and if I did I would only exploit the one I'm interested about, as you guessed it's almost always 1 price / 1 picture.
• Once home, I open my laptop ( 12-inch screen, so who's got the smallest one ? 🤣 ) , copy the pictures from the gallery to a specific folder, open the first picture with an image viewer ( it loads all the pictures in that folder, so I can easily navigate the pictures of the day )
• I also open a .txt file where I have most of my tracked products ( 1 line containing a short description and the EAN separated by several tabulation - ballpark 200 lines for products only interspersed with notes - somehow match my count of 183 displayed in my dashboard ) , ctrl+f of either the tail of the EAN or some character chain I know will give me the product I'm looking for so I can copy/paste EAN straight into the appropriate box when needed.

I'll try your method ( phone to upload pic + laptop to add the price(s) ) next week and time myself, see if it provides a sizeable improvement and report back.

PS: I have prepared a longer version ( also for myself ) detailing the specific reasons for 1 pic / 1 price, associated with a sample of a few dozens of pictures illustrating why a 1 pic / several prices, I think wouldn't work for me, I'm willing to send it to you on your email, if you want us to dig deeper.

[raphodn]

don't have much time to answer right now, but I'd be willing to dig deeper.

In a single shop & day session, how many pictures do you take ?

If there existed a multi-upload proof page (similar to https://prices.openfoodfacts.org/proofs/add/single - yeah needs to be fixed - but allowing to select multiple images for the same shop & date) would it help you ? though afterwards you would need to go to your dashboard > proofs > for each one add the corresponding price ^^ so only a marginal time gain.. ?

[raphodn]

@serpico

No worries, I took notes so we can postpone for later, whenever...

I'd say 15 to 40 pictures, not all will be uploaded ( doubled for quality )...but I'd say 10-12 prices per session minimum and as many pictures.

If there existed a multi-upload proof page (similar to https://prices.openfoodfacts.org/proofs/add/single - yeah needs to be fixed - but allowing to select multiple images for the same shop & date) would it help you ? though afterwards you would need to go to your dashboard > proofs > for each one add the corresponding price ^^ so only a marginal time gain.. ?

I think it'd help, but let's table that, first I need to try your workflow, see how it goes.

I'd really like to give you an embryo of a mock up but I need to really to decompose the process ( i.e : upload picture > copy the EAN from .txt > hit tab on OP > paste the EAN > ) I'll open another issue for me to work on that, if you could take a look at it whenever you can and tell me what you think.

Here is the issue in question 961

[serpico]

Here's some feedback after a few price logging sessions, using some of your advice regarding the workflow :

A/ First session

1/ Upload the proofs using the webapp on the phone ( same type, same folder for source image, same shop location, same date, same currency ) for all the 29 pictures, it took 20' so on average 41" per picture, including 13" really uploading ( when you hit the green button after selecting the file/image, the button is spinning and when it's over the "done" button is unlocked and you can go into the menu and start the next proof.

2/ Log The prices
I followed your workflow, once all the proofs are uploaded I go into the proof gallery, bottom menu of one proof card > Add a price

Hit "TYPE A BARCODE"
copy/paste the barcode
Hit enter
Click on the price input box ( to focus )
Type the price ( display the last known price for this product and this location and openfoodfacts/open-prices-frontend#567 could be useful )
Hit upload
Hit done

Here there's probably something to do in term of keyboard shortcuts, where the focus gonna be/should be every time we validate each step, there's probably a few keystroke, pointer movement to be saved...

19 prices in 18'42" or 1122" 59" per price

So combined, per proof+price, it averages 1'40" per price ( upload and logging ).

This is, what I consider, a significantly favorable sample as I trimmed my .txt file containing the EANs so I only have products from this shop and sorted in the order I go through the shop - there's a caveat here, the order in which I upload the proof might have an slight impact, it'd better to have the same order on the .txt file and on the proof gallery. I also keep the new products ( the ones that are not on my EAN trimmed list/.txt file ) as process separately - and it takes a lot more time ( change the quantity on OFF, add the brand, create the product from scratch...), but it's significant in my case, I'd say for every 15 to 20 known product ( allow fast mode, copy paste EAN and that's it ) there are 3 to 5 which require the slow way, I have a ballpark somewhere, I'll try to look for it

B/ Second session

1/ I copied the proof pictures from the phone to the laptop in a dedicated folder, then upload the proofs using the browser of the laptop instead of the doing on the phone like in A/.
There's a noticeable improvement to upload x16 proofs it took 8'44" or 524" or 33"per upload ( this is including the step to create the folder in the laptop, copy from phone to laptop )

2/ log the prices : x13 prices in 12'17" soit 737" ou 57" per price

Combined 1'30" per proof+price ( upload and logging )

C/ Third session

1/ Same method as in B/ ( copy pictures from phone to laptop, then upload from laptop )
upload 17 proofs ( some duplicates ) in 8'20" or 500" averages to 29" per proof

2/ log the prices : 11'40" for 11 prices or 700" averages 64" per price

Combined 1'33" per proof+price ( upload and logging )

One thing that still slows the process, is once the price is sent the dashboard page is automatically loaded, then you need to click the Proof button at the end of each price, load the gallery, look for the next proof - if we went back directly to the proof gallery we could save some time - even more if there was an option to display only the proofs with 0 price (having a "stack/deck" of proofs to go through for one session, the stack would slim down every time we log price - only proofs with 0 price would stay in the "stack/deck" to be exploited - so if you mess up the upload order compare to your list of EANs and proceed in reverse order, each time you need to load 1 or more block of X proofs - a more desirable situation would be the first ( at the top ) proof you load/see in the proof gallery is the one associated with the next EAN on your list.

Hope it's not too messy.

[TTalex]

Related: I've noticed losing a bit of time when mistyping/misreading barcodes, so I've proposed a solution here: openfoodfacts/open-prices-frontend#991

Note that this doesn't really help on the above feedback. It might even worsen it, since it breaks some of the enter key shortcuts.

[raphodn]

Can you explain a bit more ? You mistype/misread the barcode because the image is not very clear, or your hesitating between a 3 & a 6 or 8 for instance ? (edit: ok read your PR description, there's more details there 👍 )

I opened an issue in search-a-licious a few weeks ago to see what could be done for the fuzzy search, probably related :)
openfoodfacts/search-a-licious#242
--> these complex product barcode queries could be sent to search-a-licious instead of OP backend (but we wouldn't have the price_count)

[raphodn]

With the recent "Contribution assistant" built by @TTalex , I'm seeing 2 (or 3) seperate steps that we could isolate, allowing distribution on multiple users.

Some of these thoughts have already been shared in a similar discussion - #523 - and some related issues.

Step 1: upload 1 or multiple proofs [user]

Speed up proof addition when a user has multiple proofs for a given location & date.

Related issues

• Multiple proof upload page open-prices-frontend#983

Step 2: extract data from images [AI/ML]

• automatically predict the proof type
• automatically crop out price tags (bounding boxes)
• ask an AI to return the price predictions (see the current Contribution assistant that uses Gemini AI)

Related issues

• Proof : auto classification on user upload #394
• Proof: send to Google Cloud Vision, and store the matching JSON output #320
• AI-powered price addition workflow (Contribution assistant experiment) open-prices-frontend#1044
• Create a proof_prediction table to store predictions from ML models #511
• Add a price_tags table to store predictions #611

Step 3: ask users to validate the price predictions [user]

Last step, we have price info that has been extracted from the images, we need a user to validate and make sure that it is the correct product (or raw category) and price.

Related issues

• Proofs (Price tags): allow users other than the owner to add prices #568

[raphodn]

Quick update :

1. in the frontend the multiple proof upload page is live since a few days

• https://prices.openfoodfacts.org/proofs/add/multiple
• it's missing a check on the images (and some info message) to make sure they are all from the same date

2. in the backend we've unlocked 2 things

• proofs can now be uploaded anonymously
• any user can add prices on PRICE_TAG proofs

Next step:

• frontend: a new contribution page displaying all PRICE_TAG proofs without any prices. And allowing any user to start adding prices (decided to move faster and go directly to the price-level instead of proofs)
• frontend: start splitting your price dashboard, between personal data (RECEIPT/GDPR proofs & their related prices), and common data (PRICE_TAG proofs or prices)
• backend: store price-level predictions
• frontend: a new contribution page displaying price tags to be validated (and effectively create prices from predictions)

[raphael0202]

Agree! I would also add as next step, add a multiple image upload form accessible without log-in for anonymous proof upload (as discussed last wednesday).

[raphodn]

fyi i updated the workflow, to show the intermediate "price tag" prediction

[raphodn]

Most of the work has been done in the backend during the last few days !!
#611

[raphodn]

I just created an issue in the frontend to start working on a "price tag validation" UI
(restricted to only your proofs/prices for now)
openfoodfacts/open-prices-frontend#1137

[serpico]

Should we split the discussion ? as :

• Speed up proof & price addition using AI
• Speed up proof & price addition by refining the UI [ full manual - one of the scenario I'm thinking about is the picture is so bad the person that took it can barely make out the EAN ( still can but also by trying to amend/correct the EAN and cross reference/check using brand, shop, whatever ) so what's AI gonna do...]

Example : In the above workflow I'd, personally, would put "Classify receipt/price tag" on the 1st step since when I have a batch of price/pictures to upload I do them all, without receipt in between...

[raphael0202]

Even if the EAN cannot be read, we can still provide product suggestion based on :

• the price tag label (ex : confiture mûre 400g Grosjean)
• the product that is associated with the price tag: we can find similar products in OFF based on the product image

We can't guarantee a 100% match rate, but at least we could provide valuable suggestions most of the time.
When ML fails, we should always let the user provide the data efficiently. ML-based contribution and manual contribution should be thought as a whole in my opinion.

[bobanmilano]

sorry, i'm not sure if my question is on the right place here. i tried for the first time to add a price but found out that i had to upload a "proof" image first. so i made the image (jpg, ~50KB) and tried to upload it via POST to https://prices.openfoodfacts.org/api/v1/proofs/upload.

but i get a 500 server error each time. is the server not working correctly or am i doing something wrong here?

[raphodn]

Heads up:

• proofs are considered valid if they are photos of price tags in stores, or photos of receipts (or GDPR requests of fidelity cards) (or authorized import of shop). The first proof you uploaded of Nutella, without any shop info nor currency, and taken from the internet, will be flagged as invalid once we have a moderation mechanism (soon)
• we highly encourage having a location for each proof. Without, well the data will be inexploitable / of lower quality
• if you want to test the API, there is a pre-prod environment : https://prices.openfoodfacts.net
• are you on Slack ? There is a #prices channel
• if you have any questions or struggle or lack of docmentation, tell us, we haven't prioritized it much yet..

How did you get to know the project btw ?

[bobanmilano]

yeah, thanks for the invitation. i would love to join your meeting at 2pm on thursday!

@own backend: only something very basic, probably just firebase to handle useraccounts (login, registration ....)
@openfoodfacts: i know it's the same userbase, i once made a simple "product information app" using openfoodfact-API (only read, no write)
@price stats: we don't need that. it's really only about comparing prices in ATD and DE and post the result on social media, very simple. i am doing this in my free time, so no big time and ressources investment - just a simple app that can be made and launched within a couple of days.
@Proofs: as soon as the app is in use there will be - hopefully - lot's of austrians living in the border region to germany who will be creating real price updates from real shops and making the photo of the product and price within the app
@documentation: thank you very much, i really appreciate your support. there are questions all the time :) like e.g. now somehow the display of the "proof" image is not working. i also tried to find the #prices channel on slack but it could not be found. do i need an invitation for that channel?

i found "openprices" when i searched the internet for a way to fetch prices from germany. but there is no easy way to do that. the shop chains are technically doing everything that's possible to even prevent webscraping and they have no public database with prices. so i found on the internet the "openprices" project and it would solve perfectly the goal to compare prices (as soon as enough prices from germany and austria are available).

the difference in prices between austria and germany is up to 100% and this fact is these days discussed very heavily in politics. even austrian products are cheaper in germany than in austria - can you believe that? 😅

so, i hope to make some change in this whole unfair situation by enabling the customers to put some presser on the shop chains by posting their prices on social media and flooding their email-accounts with customer protests. 😁

[raphael0202]

@bobanmilano The link of the #prices Slack channel: https://openfoodfacts.slack.com/archives/C02Q6KM7F

Regarding the proof image, can you display the URL of the image you try to display? And do you have a status code by any chance?

[bobanmilano]

i have decided - considering all the overhead and additional steps for the app-user when working with openprices - to use firebase as a way to store prices. the reason is: my app does not need a 100% proof where the price is found and so on. it's really only about recording price differences (which everyone can also find in german and austrian webshops and be able to record them in the app) and post them on social media.
the "time window" for such an app is very short as it is currently a very hot topic here up to the highest levels of politics. in 2-3 months the topic will be probably completely forgotten.
so the "time question" and the simplicity for the user are the main reasons for deciding do use a custom backend. but thanks anyway for your kind support. i think openprices and openfoodfacts are great projects and i wish you all the best!
kind regards
boban

[raphael0202]

I understand, good luck with your project! If you happen to develop a more long-term solution for price tracking, we will always welcome contributions from external apps :)