Download and archive linked websites

[datokrat]

As a person who often references blog posts and the like, it sometimes happens that links turn dead after a while. At the moment, I have the following options.

• risking losing the linked website
• hoping that archive.org will store the website and looking it up when a link turns dead
• backing up links by hand and copying local links into the knowledge base.

I imagine a world to be really cool in which my knowledge base handles this for me. More precisely,

• when I create a HTTP or HTTPS link, Athens requests the target page and saves a HTML backup in ~/athens/websites/
• there is a possibility to open the backup instead of the actual link (maybe using a context menu or some special link syntax telling Athens to link to the corresponding backup?)
• this feature can be (de)activated in the settings
• there is an option in the settings to prune the backed-up websites (i.e., removing all orphaned backups to which no link exists any more).

Another benefit of this feature would be that one would be able to visit the snapshot of the linked website as it was when the link was created.

However, there are some complications or open questions that we would have to sort out:

• How to handle JavaScript-heavy pages like Twitter? It's -- probably -- unrealistic to execute the JavaScript code in a headless browser environment and take store the DOM after some fixed timeout. Furthermore, Cookie consents and the likes might hurt a lot.
• When should a snapshot be renewed? Imagine that the bloc containing the link is edited. Does this mean that the old snapshot should be replaced through a new one?

[tomisme]

Might be worth looking into https://github.com/ArchiveBox/ArchiveBox

[tangjeff0]

How to handle JavaScript-heavy pages like Twitter? It's -- probably -- unrealistic to execute the JavaScript code in a headless browser environment and take store the DOM after some fixed timeout. Furthermore, Cookie consents and the likes might hurt a lot.

There are quite a few Twitter bookmarking services. I imagine they use the Twitter API. So maybe not an elegant generic solution for all websites.

When should a snapshot be renewed? Imagine that the bloc containing the link is edited. Does this mean that the old snapshot should be replaced through a new one?

Seems like the job of an archive tool to think about these things? #1160 (comment)

when I create a HTTP or HTTPS link, Athens requests the target page and saves a HTML backup in ~/athens/websites/

I imagine there's an Emacs plugin that does something like this.

@datokrat

[datokrat]

Thanks for your thoughts! I agree that there already exist solutions for this problem and Athens should focus on other things. 👍

[zhangxiubo]

I usually convert the webpage I want to archive into a single-page PDF using https://www.sejda.com/html-to-pdf, then just upload the PDF wherever needed.