Better export method ?

[ekianjo]

Hi, not a feature request per se, but I am wondering if one decides to change from sqlite to an actual db later on, what would be the appropriate way to do it without losing data? Could this be documented in the wiki at some point?

[bvp]

@ekianjo Hi
Why not use export/import?

[ekianjo]

@bvp I am fine using that, if it's confirmed to work out of the box. Just wondering if this is tried and tested?

[RadhiFadlillah]

@ekianjo this is a good point, actually.

Right now when switching database all data will not be moved to the new database. And, unfortunately export is not really suitable because it will only give Netscape Bookmark file which only contain title, URL and tag. So, the excerpt, cached content and bookmark's ID (which is vital for specifying the thumbnail and archive) won't be exported.

With that said, I think the best direction is to split export to several sub commands. This way, we can also add other export format like static files as specified in #40.

I guess the command might look like this :

$ shiori export -h

Export bookmarks into another format

Usage:
  shiori export [command]

Available Commands:
  netscape      Export into Netscape Bookmark File
  sqlite        Export into SQL file with SQLite syntax
  mysql         Export into SQL file with MySQL syntax
  postgresql    Export into SQL file with PostgreSQL syntax
  static        Export into static files

Flags:
  -h, --help   help for export

Global Flags:
      --portable   run shiori in portable mode

[ekianjo]

Sounds like a good solution.

[deanishe]

If the idea is to re-import the file, I think JSON would be a better solution.

Exporting SQL is great for backup purposes, and every RDBMS can easily export its own flavour of SQL, but it'd be tricky to export MySQL-flavoured SQL from a SQLite database, for example.

I would suggest:

netscape    Export Netscape bookmark file
sql         Export SQL file for current database
json        Export Shiori JSON
archive     Export all Shiori data as a .tgz file

Whereby Shiori can also import any of the formats, but the SQL would only be compatible with the same kind of database it was exported from. The other formats would be database-agnostic.

[ekianjo]

@deanishe so how do you import a SQL export ? Is there a SQL export format that works for all kinds of RDBMS?

[deanishe]

so how do you import a SQL export ?

You pass it straight through to the database engine. It's probably not worth the bother implementing in your own application, however, as every RDBMS comes with tools for backing up & restoring databases, and it's literally a one-liner.

Is there a SQL export format that works for all kinds of RDBMS?

No. Each RDBMS only really speaks its own dialect of SQL, and you can't get one to emit SQL that another understands without (often complex) pre-processing.

As Shiori already knows how to convert its data models to/from JSON, it's probably easiest to use JSON(lines) as the intermediate format for migrating from one database to another.

[jsntn]

Thanks so much for this great application!

Just be here and see this discussion as I also would like to raise feature request regarding exporting to static files like .html.

Please allow me to ask if any update and plan on this?

Thanks and regards,
Jason

[fmartingr]

Hey @jsntn, you mean exporting the readable and archive versions to HTML? Regarding the archives I agree with @deanishe, we should do an intermediate format that only shiori knows how to process to the various database engines, but a proper export should also contain the archives as well, so I was thinking a zip file with a JSON file for the contents of the database and a folder with all the archives. Is this what you want or do you also request the option to export individual archives to HTML files?

[jsntn]

Hi @fmartingr thanks so much for the reply! I mean exporting the individual archives to HTML files. Actually I think they are two feature requests. Please let me explain my understanding here.

1. @deanishe 's idea is good and based on re-importing the data in terms of shiori backup and restore. In this regard, I agree with what you and @deanishe shared.

2. If it is for the purpose of full offline archive without shiori dependency, then the .html files would be a good option I think, as it is able to be read by using any internet browser across different platform without shiori, but not an intermediate format that the end-users need to use shiori to parse and process.

Very personal opinion, and I am open to discuss, please.

By the way, when you mention the 'archives', I understand they are eventually the files in archive folder of shiori-data, right? those files are actually some kind of binary ones that are only visible to shiori? As I tried to open it with internet browser but failed…

[fmartingr]

If it is for the purpose of full offline archive without shiori dependency, then the .html files would be a good option I think, as it is able to be read by using any internet browser across different platform without shiori, but not an intermediate format that the end-users need to use shiori to parse and process.

With this, do you want the archives of the bookmarks or a full offline copy of shiori? I mean, the bookmark archives and some sort of index or browseable list of all archives?

By the way, when you mention the 'archives', I understand they are eventually the files in archive folder of shiori-data, right? those files are actually some kind of binary ones that are only visible to shiori? As I tried to open it with internet browser but failed…

Those files are made with warc. We need to replace that with obelisk at some point, which creates self-contained html files.

[jsntn]

With this, do you want the archives of the bookmarks or a full offline copy of shiori? I mean, the bookmark archives and some sort of index or browseable list of all archives?

Great, that will be definitely helpful!

Then the exported package for restoring shiori can be like,

  |-- index.html
  |-- archive (folder)

Where the index.html is index to the warc files in archive folder, and those warc files are included in archive folder. correct?

Those files are made with warc. We need to replace that with obelisk at some point, which creates self-contained html files.

Thanks for sharing this! I see that is a CLI tool for saving web page as single HTML file, and from the README it seems that cannot be used to convert its output to static .html files?

Just curious to try to rename the files 4 in archive as 4.warc and open it by using https://replayweb.page/ but no response. 😄

[fmartingr]

As you can see there's no time to work on this at the moment, but in my head this would work by providing several exporter/importers when doing the actions. We could have one internal format to use ourselves for backups and other one that could be a "readable" version of the bookmarks like you propose, that export to HTML files so it can be opened and read directly into a browser.

I may open a meta issue to start discussing how the feature would eventually look like and start planning and creating tasks for the future.