Fix: for duplicate education entries#254
Open
syed0596 wants to merge 3 commits intojoeyism:masterfrom
Open
Conversation
fixed handling for when a person has multiple positions under a company
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I noticed that when scraping a profile, the
get_educations()method was adding the same education entry multiple times to the final list. This seems to be caused by the scraper matching multiple HTML elements for a single entry on the education details page.Solution:
This pull request fixes the issue by:
scraped_education_keysset in thePersonclass.get_educations(), it creates a unique key for each scraped entry.Educationobject if its key hasn't been seen before.This ensures that each education is only recorded once. I also restored the
scrape_logged_inmethod which appeared to have been accidentally deleted in the version I had.Thanks for your consideration!