Unauthorized esdedupe

[dipenpatel235]

i try below command and get error unauthorized.

docker run --rm deric/es-dedupe:latest esdedupe -H htttps://elastic_hostname -P 443 -i index_name \
-f md5 --user elastic_user --password "elastic_password"

[deric]

Try using the latest version:

docker run --rm deric/es-dedupe:v2.0.0 esdedupe -H elastic_hostname --noop

if you're using SSL, please use the --ssl flag:

docker run --rm deric/es-dedupe:v2.0.0 esdedupe -H elastic_hostname -P 443 --ssl -i index_name \
-f md5 --user elastic_user --password "elastic_password"

[dipenpatel235]

@deric it's the same issue..
after that i have try using below command so using that authentication done successfully but getting other error now

docker run -it --rm deric/es-dedupe:latest esdedupe -H http://myuser:mypassword@172.31.11.11 -P 9200 -i "myindex" -f source_ip,feed_provider

so its giving below error and my given key name is correct

2021-10-01T12:46:41  [140129338693440] INFO  esdedupe Building documents mapping on index: myindex, batch size: 1000
Traceback (most recent call last):
  File "/usr/local/bin/esdedupe", line 11, in <module>
    load_entry_point('esdedupe==2.0.0', 'console_scripts', 'esdedupe')()
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/cmd.py", line 67, in main
    lastcmd = ''
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/esdedupe.py", line 109, in run
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/esdedupe.py", line 169, in process_index
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/esdedupe.py", line 188, in scan_and_remove
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/esdedupe.py", line 178, in scan
  File "/usr/local/lib/python3.7/dist-packages/esdedupe-2.0.0-py3.7.egg/esdedupe/esdedupe.py", line 36, in build_index
KeyError: 'source_ip'

[deric]

That looks like different error than Unauthorized to me.

source_ip field is probably not included in all documents. Try running the command with higher verbosity:

docker run -it --rm deric/es-dedupe:latest esdedupe -H http://myuser:mypassword@172.31.11.11 -P 9200 -i "myindex" -f source_ip,feed_provider --level DEBUG --es-level DEBUG

[dipenpatel235]

Yes it's possible it's not in all all documents. .

…

On Fri, Oct 1, 2021, 7:11 PM Tomas Barton ***@***.***> wrote: That looks like different error than Unauthorized to me. source_ip field is probably not included in all documents. Try running the command with higher verbosity: docker run -it --rm deric/es-dedupe:latest esdedupe -H ***@***.*** -P 9200 -i "myindex" -f source_ip,feed_provider --level DEBUG --es-level DEBUG — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6CDOJ7NNWGTGQDLP7X46LUEW275ANCNFSM5E7A4BHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[deric]

Then you can't use it as part of an unique key. I'll try to throw a better error message.

[dipenpatel235]

Ok... So is there any other way to delete ? Because I needed this things .

…

On Fri, Oct 1, 2021, 7:32 PM Tomas Barton ***@***.***> wrote: Then you can't use it as part of an unique key. I'll try to throw a better error message. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6CDOMU4VBE4MLBWHXLIBTUEW5QFANCNFSM5E7A4BHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[dipenpatel235]

@deric if the key field does not exist that I have fixed using the below method.
modified esdedupe.py line no 36 to below code. you can add this code block into your code and push it.

                try:
                    combined_key += str(hit['_source'][field])
                except Exception as e:
                    pass

[deric]

@dipenpatel235 That's not really a fix, rather a dirty-bomb which might lead to deleting all data from an index, if incorrect field name is used.

[dipenpatel235]

Yes right... But something you can put restrictions on that if we can find better way.. But as of now I have did that for my custom solution.

…

On Mon, Oct 4, 2021, 6:03 PM Tomas Barton ***@***.***> wrote: @dipenpatel235 <https://github.com/dipenpatel235> That's not really a fix, rather a dirty-bomb which might lead to deleting all data from an index, if incorrect field name is used. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6CDOLWDWAWEL2PJ7BNRX3UFGNLDANCNFSM5E7A4BHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[adhaamehab]

I had the same issue when passing --user and --password, the workaround is to pass the host as
https://user:password@host:port and pass --ssl