immich-go not uploading most of my 202Gig Takeout

[phands]

Hi all,

Sorry if this is a daft question.
I have a large-ish (just over 202Gigs) Takeout directory (extracted from the compressed archives to a directory structure), and I'm trying to use immich-go to upload it all.

It seems to work partially, but I only see a tiny fraction of the files in the immich server. There are tens of thousand of photos and videos. but less than 25 in Immich.

I know that the uploader is changing rapidly, and I'd love to help with testing if I can.
My Takeout directory is structured as shown in the screenshot...

Immich-go says it's seeing 31553 images...see this screenshot...

my iimmich-go command is ....
/immich-go upload from-google-photos /local-12T/Google\ Photos/Takeout --api-key 853oJrfwOpiGB8XhASoAraVsQ8USDS156Eu1nLaNWZY --server http://10.0.0.37:2283

The log file seems OK, but shows that while it scanned 31553 images, it only uploaded 35 the first time, and now they are duplicates. Here are the last few lines of 102900...

2025-01-24 12:31:34 INF Tagged file=Takeout:Photos from 2024/Simon Durrant and Anna and Alan Kidd and wife.jpg tags=[Takeout]
2025-01-24 12:31:34 INF added to an album file=Takeout:Photos from 2024/Simon Durrant and Anna and Alan Kidd and wife.jpg Album=TATC 11th Entry
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Input analysis:
2025-01-24 12:33:39 INF ---------------
2025-01-24 12:33:39 INF scanned image file : 31553
2025-01-24 12:33:39 INF scanned video file : 2926
2025-01-24 12:33:39 INF scanned sidecar file : 33311
2025-01-24 12:33:39 INF discarded file : 0
2025-01-24 12:33:39 INF unsupported file : 468
2025-01-24 12:33:39 INF file duplicated in the input : 7
2025-01-24 12:33:39 INF associated metadata file : 42
2025-01-24 12:33:39 INF missing associated metadata file : 34437
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Uploading:
2025-01-24 12:33:39 INF ----------
2025-01-24 12:33:39 INF uploaded : 0
2025-01-24 12:33:39 INF upload error : 0
2025-01-24 12:33:39 INF file not selected : 0
2025-01-24 12:33:39 INF server's asset upgraded with the input : 0
2025-01-24 12:33:39 INF server has same asset : 35
2025-01-24 12:33:39 INF server has a better asset : 0
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Check the log file: /home/phands/.cache/immich-go/immich-go_2025-01-24_12-31-01.log

Can anyone point me at where to start debugging or what is wrong with my immich-go command? I'm also available for live debugging (USA Eastern time).

Thanks,

Paul

[simulot]

The JSON files have an unusual name:

photo.jpg-supplemental-metadata.json 

This explain this missing associated metadata file : 34437

You're the first to report such case.
Could you grab the list of files? See the page https://github.com/simulot/immich-go/blob/main/docs/how-to-send-debug-data.md

The takeout isn't easy to parse. Long file names are truncated using unclear rules. So, I'm surprised to have much longer name for the JSON files. I'm particularly curious how are named JSON files attached to photos with very long names...

I have requested a fresh takeout, to see if something has changed.

[phands]

Here's another view of the list...

list2.txt

[simulot]

Thank you for the list. This is new for me.

Some weird cases

Screenshot_20231027_123303_Facebook(1).jpg
Screenshot_20231027_123303_Facebook.jpg.supple.json
Screenshot_20231027_123303_Facebook.jpg.supple(1).json
Screenshot_20231027_123303_Facebook.jpg
Screenshot_20231027_123303_Facebook-edited.jpg
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_.json
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_2.jpg
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_2(1).jpg

I'll add the support for those case very soon

[simulot]

Issue #652

Could you provide the content of some of the supplemental-metadata.json files ?

[phands]

I'll get on it today...do you want the metadata for the particular files you list above, or just any metadata?

[simulot]

any metadata is good.

[heylookltsme]

Hello! I'm having this same issue. Must be a recent change with Google Takeout because I first started playing with immich-go a couple weeks ago and everything worked as expected. But a couple days ago I wiped my immich installation to start fresh and used a new takeout export and now I'm seeing this exact issue.

Here's the summary of a dry run:

Input analysis:
---------------
scanned image file                      :   11990
scanned video file                      :     453
scanned sidecar file                    :   14386
discarded file                          :       0
unsupported file                        :      16
file duplicated in the input            :      14
associated metadata file                :      75
missing associated metadata file        :   12368

Uploading:
----------
uploaded                                :      61
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :       0
server has a better asset               :       0

Here's the contents of one of the supplemental-meta.json files:

{
  "title": "PXL_20221003_181216799~2.jpg",
  "description": "",
  "imageViews": "1",
  "creationTime": {
    "timestamp": "1670067700",
    "formatted": "Dec 3, 2022, 11:41:40 AM UTC"
  },
  "photoTakenTime": {
    "timestamp": "1664820736",
    "formatted": "Oct 3, 2022, 6:12:16 PM UTC"
  },
  "geoData": {
    "latitude": 43.190594399999995,
    "longitude": -75.1767833,
    "altitude": 301.33,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 43.190594399999995,
    "longitude": -75.1767833,
    "altitude": 301.33,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "people": [{
    "name": "Kinsey"
  }, {
    "name": "Christie Seaver"
  }],
  "url": "https://photos.google.com/photo/AF1QipM4SRG7vwLtVXUeBw2hUz1Hu39sXQ7sBhlZu0JP",
  "googlePhotosOrigin": {
    "mobileUpload": {
      "deviceFolder": {
        "localFolderName": ""
      },
      "deviceType": "ANDROID_PHONE"
    }
  }
}

Please let me know if there's any other information that would be helpful for you. Thanks for your great work on this project!

[simulot]

I'll release a new version in a couple of hours

[heylookltsme]

Things look much improved with the latest version! Thanks much!!

Input analysis:
---------------
scanned image file                      :   55837
scanned video file                      :    4022
scanned sidecar file                    :   59818
discarded file                          :       8
unsupported file                        :      47
file duplicated in the input            :   29851
associated metadata file                :   59674
missing associated metadata file        :     181

Uploading:
----------
uploaded                                :   29819
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :       0
server has a better asset               :       0

Are these results expected? I'm trying to figure out how the numbers add up.

Is this what I should expect?

scanned images + scanned videos - discarded - unsupported - duplicates - missing metadata = files uploaded

If so, the numbers don't seem to add up so I think I may be misunderstanding. I just want to make sure nothing is missed. Thank you!

[simulot]

I try to do the math correctly... but it's more challenging than expected. some problems are counted twice and some CLI options may change the behavior. unsupported aren't image or video.

[phands]

Thank you! As soon as it's available, I'll test it! Paul

…

On Mon, Feb 3, 2025 at 1:46 PM Jean François CASSAN < ***@***.***> wrote: I'll release a new version in a couple of hours — Reply to this email directly, view it on GitHub <#650 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCC5NBRJXC5QQ6VXB5PDED2N62RLAVCNFSM6AAAAABV2FPVYWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBUGYYTMNQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[simulot]

you may haven't noticed I have released a new RC this morning.

[phands]

Hi, The new version worked very much better! Thank you!! I attach 3 screenshots. The first is a view of the finished run, with the statistics.... [image: immich-go successful run.png] As you can see, it now uploaded a lot more files - 27156 as opposed to 35, which is great! However, it still seems that there are a bunch of files that didn't get uploaded. Is there a log somewhere that I can use to see what didn't make it, and perhaps why? I'd love to help as much as I can. The other 2 screenshots are just background material... This is the view from "top", showing that gunicorn and immich use a fair amount of CPU during the upload, but immich-go is not bad at all.[image: immich-go top screenshot.png] I'm not sure how lower power machines would cope, but for me, it's OK, as my machine is reasonably high-spec for a home box... Operating System: KDE neon 6.2 KDE Plasma Version: 6.2.5 KDE Frameworks Version: 6.10.0 Qt Version: 6.8.1 Kernel Version: 6.8.0-52-generic (64-bit) Graphics Platform: Wayland Processors: 24 × AMD Ryzen 9 3900X 12-Core Processor Memory: 125.7 GiB of RAM Graphics Processor: AMD Radeon RX 6600 Manufacturer: ASUS The third screenshot is a view of iotop, showing that the network demands of the api and immich are quite reasonable![image: immich-go iotop screenshot.png] Anyway....once again, thanks for Immich, and I am available to help test and debug - just ask. Paul

…

On Tue, Feb 4, 2025 at 1:52 PM Jean François CASSAN < ***@***.***> wrote: you may haven't noticed I have released a new RC this morning. — Reply to this email directly, view it on GitHub <#650 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCC5NCOWYLFIRNFKQTQGJD2OEEARAVCNFSM6AAAAABV2FPVYWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBVHA2TMMY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[simulot]

The screenshots aren't visible.

Immich-go parses the source folder and queries immich about existing assets. The heavy lifting is done by Immich, all photos are resampled, and IA task performed.

I have received positive feedback from users using a raspberry Pi4 for immich. But the feeling is: be patient...

Yes, there are log files.
Linux $HOME/.cache/immich-go/immich-go_YYYY-MM-DD_HH-MI-SS.log
Windows %LocalAppData%\immich-go\immich-go_YYYY-MM-DD_HH-MI-SS.log
macOS $HOME/Library/Caches/immich-go/immich-go_YYYY-MM-DD_HH-MI-SS.log

You can share the log, for investigation.

[phands]

Hi Jean François,

Again, thanks for Immich...it's excellent!

I have no idea why the screenshots sign't show - I see them embedded in the GMAIL. but they are really only for info, and I see that the summary info as at the end of the logfile.....

2025-02-04 19:50:15 INF Tagged file=Takeout:Weekend in Millerton/20160730_162025.mp4 tags=Takeout
2025-02-05 09:50:32 INF 
2025-02-05 09:50:32 INF Input analysis:
2025-02-05 09:50:32 INF ---------------
2025-02-05 09:50:32 INF scanned image file                      :   31553
2025-02-05 09:50:32 INF scanned video file                      :    2926
2025-02-05 09:50:32 INF scanned sidecar file                    :   33311
2025-02-05 09:50:32 INF discarded file                          :       0
2025-02-05 09:50:32 INF unsupported file                        :     468
2025-02-05 09:50:32 INF file duplicated in the input            :    5708
2025-02-05 09:50:32 INF associated metadata file                :   33230
2025-02-05 09:50:32 INF missing associated metadata file        :    1249
2025-02-05 09:50:32 INF 
2025-02-05 09:50:32 INF Uploading:
2025-02-05 09:50:32 INF ----------
2025-02-05 09:50:32 INF uploaded                                :   27156
2025-02-05 09:50:32 INF upload error                            :       0
2025-02-05 09:50:32 INF file not selected                       :       0
2025-02-05 09:50:32 INF server's asset upgraded with the input  :       0
2025-02-05 09:50:32 INF server has same asset                   :     366
2025-02-05 09:50:32 INF server has a better asset               :       0
2025-02-05 09:50:32 INF 
2025-02-05 09:50:32 ERR Error tagging assets with tag Tag=Takeout error=BulkTagAssets, PUT, http://10.0.0.37:2283/api/tags/assets, 500 Internal Server Error
Failed to bulk tag assets

Searching for "Error" in the logfile shows 40 hits

The log file is 21M in size, and I see 40 hits on a simple search for error. They all appear to be the same error - here's one example...

2025-02-04 19:49:52 INF Tagged file=Takeout:Untitled(455)/20160911_154630.jpg tags=Takeout
2025-02-04 19:49:52 ERR Error tagging assets with tag Tag=Takeout error=BulkTagAssets, PUT, http://10.0.0.37:2283/api/tags/assets, 500 Internal Server Error
Failed to bulk tag assets

Hope this helps,

Paul

[phands]

Here's a full copy of the log file, just for completeness.

immich-go_2025-02-04_15-44-09.log

[larryhowe]

First, I am thoroughly impressed with both immich, and immich-go. Great functionality, great engineering, and maybe I shouldn't say this but so far I haven't hit anything I would call a bug. You guys have done a fantastic job.

I have never used git before, so pardon me if I don't know the customs. I have (and roughly fixed) a similar problem. In my case, I have many files that were uploaded to gphotos with the same name. So e.g. CLIP0001.AVI. In the takeout, there are now named CLIP0001(1).AVI, CLIP0001(2).AVI, etc. But, the jsons are named CLIP0001.AVI.supplemental-metadata(1).json. immich-go does not find a match for these. I created a crude script to fix the filenames, and it seems to have worked! Here is the script. It's my first bash script in probably 20 years so I'm sure it could be done better.

PS I think the reason for the duplicate filenames, is I was using a camera with an SD card, and I would d/l the pics and then wipe the card, and the camera would start over on the naming.

PPS the reason I branch the code for each extension, is that the number of dots in the filename changes. So there could be CLIP0001(1).AVI, but also CLIP0001.800(1).AVI. Rather than go crazy with regexp, I just hardcoded the extensions, and it has handled all the cases so far.

#!/bin/bash

# Set the directory where the files are located
directory="./Takeout/Google Photos/Photos from 2011"

# Loop through all files in the directory
for file in "$directory"/*.json; do

  # Check if it's a file (not a directory)
  if [ -f "$file" ]; then
    # Get the current filename
    filename=$(basename "$file")
    
    # Create the new filename
    if [[ $filename == *.mp4.* ]]; then
      new_filename=$(echo "$filename" | sed -E 's/(.*)\.mp4.supplemental-metadata(.*)\.json/\1\2.mp4.supplemental-metadata.json/')
    elif [[ $filename == *.AVI.* ]]; then
      new_filename=$(echo "$filename" | sed -E 's/(.*)\.AVI.supplemental-metadata(.*)\.json/\1\2.AVI.supplemental-metadata.json/')
    elif [[ $filename == *.JPG.* ]]; then
      new_filename=$(echo "$filename" | sed -E 's/(.*)\.JPG.supplemental-metadata(.*)\.json/\1\2.JPG.supplemental-metadata.json/')
    else
      new_filename=$filename
    fi
    
    # Rename the file
    if [ "$new_filename" != "$filename" ]; then
      mv "$file" "$directory/$new_filename"
      echo "$filename" "$new_filename"
    fi

  fi

done

[simulot]

Thanks for your input.

The JSON matcher is supposed to do similar work, but without regexp.
I plan to rework on the JSON matcher, because of #698

Could you sent a list of relevant files exactly as they are delivered in the takeout?

[larryhowe]

Here are the before-and-after for two years worth of file names. Let me know if you need anything else!

files.before.2010.txt
files.after.2010.txt
files.before.2012.txt
files.after.2012.txt

[larryhowe]

I notice that there are files also like 1282242458760-edited.jpg. There does not seem to be a matching json.

[larryhowe]

and here is my code block for adding the missing json files for the -edited images. It just copies the original json, but that's still better than having no date associated to the file

# deal with edited files e.g. CLIP0001-edited.jpg
for file in "$directory"/*-edited.jpg; do

  # Check if it's a file (not a directory)
  if [ -f "$file" ]; then

    echo $file
    # Get the current filename
    filename=$(basename "$file")
    # Figure out which json file to copy
    old_filename=$(echo "$filename" | sed -E 's/(.*)-edited(.*)/\1\2.supplemental-metadata.json/')
    # and the new filename
    new_filename="$filename.supplemental-metadata.json"
    # Copy the file
    cp "$directory/$old_filename" "$directory/$new_filename"
    echo "$old_filename" "$new_filename"

  fi

done

[simulot]

Thanks for sharing the logs...
What's the difference between before and after files?

I quick glance to your lists. I have spotted following schemas:

• the classical regular:

  • IMAGE.jpg
  • IMAGE.jpg.json

• the new regular:

  • IMAG0440.jpg
  • IMAG0440.jpg.supplemental-metadata.json

• duplicates name in the same directory:

  • Scan35(1).jpg
  • Scan35.jpg.supplemental-metadata(1).json
  • CLIP0001(10).AVI
  • CLIP0001.AVI.supplemental-metadata(10).json

• but also 🤯 different way to process the duplicates

  • VIDEO0036(1).mp4
  • VIDEO0036(1).mp4.supplemental-metadata.json
  • VIDEO0036.mp4.supplemental-metadata(1).json

• too long to fit

  • army.blue.v.steelers.2012.11.10.4.mp4
  • army.blue.v.steelers.2012.11.10.4.mp4.suppleme.json
  • dad.snowboard.2012.02.mp4
  • dad.snowboard.2012.02.mp4.supplemental-metadat.json
  • NEMO_MOVIE_TICKET_STUB.jpg.supplemental-metada.json

Immich-go must cope with all variations of takeout structures. at the same time.

[larryhowe]

BEFORE is the file names from GP. AFTER are the file names after I ran my script to munge the filenames and copy some of the json files to fill in the blanks. What a mess google has created here. "supplemental-metadata" seems completely unnecessary and then causes problems with the length. I don't know if there are any in my data, but probably too long and duplicate name can happen together also 🥺

…

On Wed, Feb 12, 2025 at 11:37 AM Jean François CASSAN < ***@***.***> wrote: Thanks for sharing the logs... What's the difference between before and after files? I quick glance to your lists. I have spotted following schemas: - the classical regular: - IMAGE.jpg - IMAGE.jpg.json - the new regular: - IMAG0440.jpg - IMAG0440.jpg.supplemental-metadata.json - duplicates name in the same directory: - Scan35(1).jpg - Scan35.jpg.supplemental-metadata(1).json - CLIP0001(10).AVI - CLIP0001.AVI.supplemental-metadata(10).json - but also 🤯 different way to process the duplicates - VIDEO0036(1).mp4 - VIDEO0036(1).mp4.supplemental-metadata.json - VIDEO0036.mp4.supplemental-metadata(1).json - too long to fit - army.blue.v.steelers.2012.11.10.4.mp4 - army.blue.v.steelers.2012.11.10.4.mp4.suppleme.json - dad.snowboard.2012.02.mp4 - dad.snowboard.2012.02.mp4.supplemental-metadat.json - NEMO_MOVIE_TICKET_STUB.jpg.supplemental-metada.json Immich-go must cope with all variations of takeout structures. at the same time. — Reply to this email directly, view it on GitHub <#650 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFPT6WJAGL6JVWOX5B4437D2PN2D7AVCNFSM6AAAAABV2FPVYWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMJXGYYTCNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[simulot]

Thanks,
I understand better.

I guess this is a tactic to increase the "cost" of leaving them.

[larryhowe]

Agreed. It is not how you would do it if you wanted it to be simple.

Different topic: I would contact you directly but I don't know how to do that in git. I have developed a workflow for porting a large GP instance to immich using immich-go. 20+ years of files. Would it be useful if I create a document for that?

[simulot]

Good idea!
You could add a tutorial in the discussions of the project.
I'm also available on discord @simulot

You can skip the dezip and bash script. I'll update the JSON matcher with your findings.

[heylookltsme]

I have developed a workflow for porting a large GP instance to immich using immich-go. 20+ years of files.

I would be very interested in this, thank you! Downloading the takeout files is such a pain.

[larryhowe]

Just tried 0.23, got this in the log file:

WRN missing associated metadata file file=Photos from 2023:20230512_193755-edited.jpg

I made a few other tests and this is consistently an issue. In my bash script, I addressed this by just copying the json file for the original jpg and adding -edited to the name. That seemed to work.

[simulot]

This fix for the -edited image is almost ready.

[azwillnj]

I was getting a massive amount of errors on missing json files using 0.22 then I found this thread and the new version. Just ran 0.23 and its hanging on this error, going to post the tail of the log:

2025-02-14 09:46:25 INF Tagged file=takeout-20250213T052212Z-010:Takeout/Google Photos/Photos from 2011/IMG_2802.JPG tags=takeout-20250213T052212Z
2025-02-14 09:46:25 ERR upload error file=takeout-20250213T052212Z-010:Takeout/Google Photos/Photos from 2011/IMG_2802(1).JPG error=AssetReplace, PUT, http://192.168.1.99:8088/api/assets/a963f37e-5db2-4930-9517-36e41fa15c01/original, 500 Internal Server Error
Failed to replace asset

2025-02-14 09:54:51 INF
2025-02-14 09:54:51 INF Input analysis:
2025-02-14 09:54:51 INF ---------------
2025-02-14 09:54:51 INF scanned image file : 79536
2025-02-14 09:54:51 INF scanned video file : 4758
2025-02-14 09:54:51 INF scanned sidecar file : 80212
2025-02-14 09:54:51 INF discarded file : 6
2025-02-14 09:54:51 INF unsupported file : 389
2025-02-14 09:54:51 INF file duplicated in the input : 3614
2025-02-14 09:54:51 INF associated metadata file : 79225
2025-02-14 09:54:51 INF missing associated metadata file : 5063
2025-02-14 09:54:51 INF
2025-02-14 09:54:51 INF Uploading:
2025-02-14 09:54:51 INF ----------
2025-02-14 09:54:51 INF uploaded : 45
2025-02-14 09:54:51 INF upload error : 1
2025-02-14 09:54:51 INF file not selected : 0
2025-02-14 09:54:51 INF server's asset upgraded with the input : 246
2025-02-14 09:54:51 INF server has same asset : 22933
2025-02-14 09:54:51 INF server has a better asset : 947
2025-02-14 09:54:51 INF
2025-02-14 09:54:51 INF Check the log file: C:\Users\azwil\AppData\Local\immich-go\immich-go_2025-02-14_09-37-51.log

Thoughts?

[simulot]

All crashes are due to a PUT at /api/asset
This crash the server's worker, and then connection is lost

I'm trying to reproduce this on my test system.

[azwillnj]

lmk if there's any way I can help you. Of note.. 0.23 never did this.

My environment is:
Immich running in Docker on Unraid
Windows 11 VM running on the same Unraid box
Immich unraid share volume mounted as network drive on the windows VM
import zips in the Immich share on unraid in their own folder
so your script is essentially telling immich to just unzip and move the files into another folder in the same unraid share

[simulot]

You can try to upload the takeout directly.

I miss an important information from your log: immich-go tries to replace the file IMG_20111215125136.jpg file because it believes this file is 0 byte long on the server. The replacement fails with an internal error.

The error results of a chain of events that are difficult to identify.

[azwillnj]

As I had mentioned earlier, I tried this upload already with 0.22.1 and it made it all the way through but with a ton of missing JSON files. That transfer worked the whole way through the first try. The message at the end said to try the takeout download again which I did. I have been trying 0.24 and 0.24.1 with the new set of zip files. After your last message I decided to try the original set of zip files to see if it would work on those and I'm getting a similar hang on a different file.

Just for the hell of it, I just tried with 0.22.1 again and it made it all the way through but with a ton of missing JSON files again.

[simulot]

This is due to a new way for google to name the JSON...
This forced me to deliver the v0.23 version.

You still have 189 unmatched files.
Could you share the log file? You can DM it on my discord account @simulot

The last run with v0.24.1 didn't hang. This show that the hang problem is dependent of the context of previous actions... It's quit impossible to debug

[TheNilesh]

I generated a Takeout archive a week ago and am still facing the same issue, even with the latest version. I'm happy to assist with anything needed.

$ immich-go version
immich-go version:0.24.2,  commit:f1b5b47586e042dfd383e40404c9cc27174854c3, date:2025-02-23T15:19:38Z

[simulot]

It the same issue... Takeouts are full of duplicates.
You can try the v0.24.7 https://github.com/simulot/immich-go/releases/tag/v0.24.7 that should solve server's errors

[ZoltrixGFC]

@simulot thanks again for this great tool.

I just re-ran it with version v0.24.7, results are below. Assuming I am looking at the right figures, immich-go is reporting a total 59,989 (once I remove the duplicates, discarded files, and unsupported files?):

That does appear a lot better I guess? Immich photos and videos adds up to 59,232

[simulot]

The immich-go report is relative to the input data, not relative to the files already uploaded in Immich-go.

The check should be done as following;
Input assets: DiscoveredImage + DiscoveredVideo
Processed assets: Uploaded + ServerError + NotSelected + Upgraded +ServerDuplicate + ServerBetter + DiscoveredDiscarded + LocalDuplicate + MissingAssociatedMetadata

• Input Assets:

  • Images: 57,490
  • Videos: 17,049
  • Total Input Assets: $$57,490 + 17,049 = 74,539$$ .

• Processed Assets:

  • Uploaded: 0
  • Errors during upload (ServerError): 0
  • Files not selected (NotSelected): 0
  • Server's asset upgraded (Upgraded): 0
  • Server has same quality (ServerDuplicate): 59,572
  • Server has better quality (ServerBetter): 14
  • Discarded files (DiscoveredDiscarded): 102
  • Duplicates in the input (LocalDuplicate): 14,419
  • Files without sidecar (MissingAssociatedMetadata): 432
  • Total Processed Assets: $$0 + 0 + 0 + 0 + 59,572 + 14 + 102 + 14,419 + 432 = 74,539$$ .

• Comparison: The total input assets ($$74,539$$) match the total processed assets ($$74,539$$).

Thus, the data integrity check is successful.

[apeowl]

I am using the latest release 0.27.0 and am seeing this issue as well. My takeout is in multiple zips and I am doing them one by one, in smaller chunks. The current file Input analysis is showing Images: 2946 Videos: 757 and it is done and showing Files uploaded: 1247 and Server has same quailty: 610 , Server has better quality: 5.

I just ran another chunk and it keeps stopping at 26%. Input analysis shows Images: 5308, VIdeos: 2010 and in Uploading (since I have run it four times) it says Server has same quality: 1309 and Server has better quailty: 31 and Uploading is only at 26%

[simulot]

Check the log files for error messages.
The main reason for stopping without an error message is that the supervisor has killed the process