@alex000kim If you provide a raw Github README link, it doesn't work since the page is a plain string (extract() doesn't work).
Also, if the URL is invalid, the Slackbot responds "I can't provide a response. Encountered an error: 'NoneType' object has no attribute 'lower'.
I was able to fix these issues by using the following updated code:
def is_html(content):
content_start = content.lower().strip()[:15]
return content_start.startswith("<!doctype html>") or content_start.startswith("<html>")
def augment_user_message(user_message, url_list):
all_url_content = ''
for url in url_list:
downloaded = fetch_url(url)
if downloaded is None:
return user_message
# Check if the content is HTML, then use extract() to clean and extract the main text content
if is_html(downloaded):
url_content = extract(downloaded, config=newconfig)
else:
url_content = downloaded
user_message = user_message.replace(f'<{url}>', '')
all_url_content = all_url_content + f' Contents of {url} : \n """ {url_content} """'
user_message = user_message + "\n" + all_url_content
return user_message
Please consider applying these changes to your code, or feel free to use a better solution if you know of any. I just wanted to share this with you.
Thank you so much for your great work by the way! :)
@alex000kim If you provide a raw Github README link, it doesn't work since the page is a plain string (
extract()doesn't work).Also, if the URL is invalid, the Slackbot responds
"I can't provide a response. Encountered an error: 'NoneType' object has no attribute 'lower'.I was able to fix these issues by using the following updated code:
Please consider applying these changes to your code, or feel free to use a better solution if you know of any. I just wanted to share this with you.
Thank you so much for your great work by the way! :)