Table plugin for sorting, searching, downloading, etc.

[dhimmel]

I think it would also be super spiffy to have an ability to directly export tables as CSV (kind of a wishlist feature, probably). But I hunt through SI so many times looking for machine readable tables.

Originally posted by @slochower in #209 (comment) (relocated here)

[dhimmel]

@slochower I moved your comment here for organization. It helps us find comments and resolve issues if the discussion stays centered around the original issue / title.

@vincerubinetti has implemented CSV for our hetnet search. See this example:

So I'm guessing it wouldn't be much work to implement this for Manubot.

I hunt through SI so many times looking for machine readable tables.

The CSV download button would still require a manual button press, but @slochower it sounds like a manual step to get to a machine readable form is okay with you? Also out of curiosity, does copying and pasting tables into a spreadsheet fail for you? It seems to me like tables usually copy as TSV text.

[slochower]

Ah, brilliant. For some reason I didn't even think of directly trying to copy and paste. Does seem default to TSV :)

Ideally, could scrape papers for CSVs directly to grab other peoples' data, but since TSV copy and paste works out-of-the-box now, that makes me happy. Maybe we could make an option---at some point in the future (low priority)---of converting Markdown tables to CSV files that could be served in the webpage branch. That way I could just point people there for direct programmatic access to the results of my calculations. Then again, maybe this isn't the right place for that: I have all the data/analysis stored on a separate repo and can just direct people there from the paper.

[vincerubinetti]

Button press download of csv would be easy.

I don't think it would be a bad idea to have a separate .csv exported for all tables as part of the normal manubot output. It'd almost be as if it were another asset, like an image. However, we'd have to make it clear that the .csv file wasn't being directly embedded in the html page like the images are (that is, if you make changes to the .csv, the manuscript wont be updated).

[duerrsimon]

A nice option for this could be to use the datatables plugin: https://datatables.net/extensions/buttons/examples/initialisation/export.html
Offers CSV, Excel and clipboard export.
By adding the following code all tables get transformed to interactive, searchable datatables with CSV export option. Datatables uses the standard HTML markup for this. This just requires adding some stuff to the HTML code and no modifcations to the markdown to html conversion.

head

<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.21/css/jquery.dataTables.min.css">
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/buttons/1.6.2/css/buttons.dataTables.min.css">
<script type="text/javascript" language="javascript" src="https://code.jquery.com/jquery-3.5.1.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/1.10.21/js/jquery.dataTables.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/buttons/1.6.2/js/dataTables.buttons.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/buttons/1.6.2/js/buttons.flash.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jszip/3.1.3/jszip.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.53/pdfmake.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pdfmake/0.1.53/vfs_fonts.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/buttons/1.6.2/js/buttons.html5.min.js"></script>
<script type="text/javascript" language="javascript" src="https://cdn.datatables.net/buttons/1.6.2/js/buttons.print.min.js"></script>

end of page body:

$(document).ready(function() {
    $('#example').DataTable( {
        dom: 'Bfrtip',
        buttons: [
            'copy', 'csv', 'excel', 'pdf', 'print'
        ]
    } );
} );

[dhimmel]

A nice option for this could be to use the datatables plugin

Thanks for the info! If you end up creating a manuscript that uses this, let us know, since it's often nice to see it in action.

I'm not a frontend person, but my worry would be that this depends on lot's of javascript packages... not sure if this would slow down page load time measurably?

[duerrsimon]

The difference is measurable but not that drastic (3.35 sec vs 2.19 sec). Big images have a much more dramatic influence on overall page load time. Obviously, the modifications to the DOM that are made by the Datatables plugin need a bit of time.

With DataTables
41 requests
145.46 KB / 47.76 KB transferred
Finish: 3.35 s
DOMContentLoaded: 1.25 s
load: 2.10 s

Without
28 requests
121.99 KB / 43.85 KB transferred
Finish: 2.19 s
DOMContentLoaded: 233 ms
load: 425 ms

I will share the link to the manuscript once published :)
It is currently in a private repo.

[vincerubinetti]

This isn't a bad idea for the future. Though if possible -- following the precedence of the other plugins -- I'd attempt to write my own more minimal, specifically manubot-tailored plugin that does sorting and searching before including a third party library.

Thanks for the suggestion. @dhimmel do you want to rename this issue to something like "write a table plugin for sorting, searching, downloading, etc. tables"

[dhimmel]

do you want to rename this issue to something like "write a table plugin for sorting, searching, downloading, etc. tables"

Sounds good to me. I think this issue is helpful as both an interim method for users who need this feature now and a reminder to us to write a new plugin. Suggested name is good with me.

@vincerubinetti can you rename the issue? I think you should be able to with your triage permissions right?

[vincerubinetti]

@dhimmel No I'm not able to.

[duerrsimon]

Made a quick demo using a plugin similar to datatables which does not have any dependencies like Jquery.

https://github.com/duerrsimon/interactivetables
https://duerrsimon.github.io/interactivetables/

Three quirks:

• the table caption disappears. Should be recoverable with Javascript.
• the export for paginated tables only drops the data in the current view to the csv. There is a fix for this but it requires modifying the library
• Possibly a major issue. Athenapdf seems to ignore my attempts at preventing the JS being rendered only on screen media. I am using window.matchMedia("screen").matches in the pandoc plugin here https://github.com/duerrsimon/interactivetables/blob/master/build/plugins/tabulator.html
  The resulting PDF thus contains the table rendered via tabulator(which is made using divs) and not the html table. Weasyprint in my local environment correctly ignores the js instructions and prints the table like any other table in the document.