I ❤ VisiData

[anjakefala]

A space for fond testimonials about VisiData.

Have a feature you love? Had a moment of joy with insights just falling out of your data?

Please share your stories!

[geekscrapy]

regex split is one of favorite features - allows me to carve up log files intricately!

[ajkerrigan]

I've found so much to love! Without going into the deep and rewarding rabbit hole of plugins though, here are some features that help me quickly make sense of data:

• Open anything: Throw most common data formats at VisiData (compressed or not) and it will handle them smoothly.

• Find interesting data: The Describe Sheet (Shift-I) includes a "distinct values per column" count which can help identify interesting columns. The Frequency Table (Shift-F) helps find interesting row values within a column. When combined, these two features help me focus on what's important!

• Make deeply nested data "glanceable": Since a picture is worth 1000 words, imagine I want to quickly identify patterns in a JSON API response. Here's a raw sample response from a default localstack installation:

[
    {
        "Description": "default group",
        "GroupName": "default",
        "IpPermissions": [],
        "OwnerId": "123456789012",
        "GroupId": "sg-ac4e493e",
        "IpPermissionsEgress": [
            {
                "IpProtocol": "-1",
                "IpRanges": [
                    {
                        "CidrIp": "0.0.0.0/0"
                    }
                ],
                "UserIdGroupPairs": []
            }
        ],
        "Tags": []
    },
    {
        "Description": "default VPC security group",
        "GroupName": "default",
        "IpPermissions": [],
        "OwnerId": "123456789012",
        "GroupId": "sg-0dbc7584",
        "IpPermissionsEgress": [
            {
                "IpProtocol": "-1",
                "IpRanges": [
                    {
                        "CidrIp": "0.0.0.0/0"
                    }
                ],
                "UserIdGroupPairs": []
            }
        ],
        "Tags": [],
        "VpcId": "vpc-bf15a51a"
    }
]

And here's the same data piped into VisiData:

 Description        | GroupName | IpPermissions | OwnerId      | GroupId     | IpPermissionsEgres…| Tags | VpcId        ║
 default group      | default   | [0]           | 123456789012 | sg-ac4e493e | [1]                | [0]  |             ⌀║
 default VPC securi…| default   | [0]           | 123456789012 | sg-0dbc7584 | [1]                | [0]  | vpc-bf15a51a ║

I know which one I'd rather look at!

[pigreco]

Because @aborruso makes everything easy !!!

jokes aside - shift + F for the frequency table

[anjakefala]

I am most definitely so, so grateful for him producing Italian educational materials.

💜

[anjakefala]

This is one of my favorite python tools. I heard first about it from its creator Saul Pwanson's interview on Podcast.init (highly recommended). It is a great example of a tool that satisfies a real need, doesn't try to do too much and does what it is designed for very well.

• https://www.reddit.com/r/commandline/comments/gxepzn/visidata_interactive_data_tool/ft38mfx/?context=3

[anjakefala]

I just found an awesome new tool for home office productivity and viewing tables in the terminal, @visidata!
It has some really simple but effective features (like sorting and histograms). You can install it easily through pip as it's Python based!
I'm using it to check results from code running on my work computer that I'm SSH'd in to via a terminal. It's so nice to be able to check things with a fast, flexible .csv viewer (it supports other formats too!) without downloading them to my local PC =)

• https://twitter.com/emilydoesastro/status/1240923826755702785

[anjakefala]

I just wanted to pass along a word of thanks and appreciation for VisiData. I saw it on HackerNews a few months ago but didn't crack open the tool until I had this large data cleaning exercise to complete recently. It ended up saving me no less than 8+ hours of work and was such a JOY to work with.
I love everything about how it is designed—your "everything is data and treated as a sheet" thesis is brilliant. I've always had an admiration for textpunk tools, but VisiData was the first time that using one was indisputably the more pragmatic and efficient option. Thanks for your incredible work on this.

[anjakefala]

I've done loads of data prep work this past year, and I was hoping it'd be Jupyter & Pandas that made it all easier, but it's actually been @visidata. It's a remarkably powerful and well-designed tool.

• https://twitter.com/yoz/status/1130668037663731712

[anjakefala]

If you have ever wanted spreadsheets in your terminal, the tool you were wishing for was @visidata

It does:
☑ Tabular data
☑ JSON data
☑ Pretty much any data
🐍 Python transforms
✨ Various magic

• https://twitter.com/agjhnsn/status/1101562455258329088

[anjakefala]

[VisiData is] more like if csvkit and excel had a baby. While you can certainly work it into your scripts, visidata is best, imo, as a tool to explore data. If you’ve ever felt like excel should be format agnostic and handle larger datasets, then visidata is for you.

• https://twitter.com/aboutaaron/status/1088276231299178497

[anjakefala]

I am a command line junkie tmux/vim user. I believe I came across Visidata while I was googling for a command line spreadsheet. It looked really interesting, so I tried it out. Right now I have 2,500 rows in the main table I'm looking at and each row has some larger text fields maybe 10k bytes each. I have my vd postgres:localhost..pathtopostgres command mapped to vdp so I can pull it quickly, it lists the tables, then I hit enter on the table I care about, cst to go to the status column of my table, and then F to break down how many of each status I have in my table.

[anjakefala]

Hi! Most of my day is currently spend on linguistic and cultural databases and VisiData is helpful for (a) not messing with my data (e.g., if I paste a combined Unicode character, I cannot have the software translating to a pre-combined one without at least telling me) and (b) not leaving the command-line (being able to split the terminal, copy something from VIM in one window and pasting back to VisiData in the other is a must).

[wdesilvestro]

VisiData describes itself as “duct tape for your data” and no statement could be more accurate. Whenever I find myself needing to quickly explore a new dataset, perform some quick data cleaning task, or grab a few summary statistics, I’ve found VisiData to be 10x faster than using Pandas, R, or other conventional data tools. Because it is a well-written CLI tool, VisiData is also significantly faster than exploring data in GUI applications. Excel often crashes on my computer long before I’ve already loaded the data up in VisiData and begun analyzing it.

[frosencrantz]

It's hard to pick just a few things that I love about Visidata. Two of my favorites are everything is a sheet and the selection mechanism. Just like in Unix everything is a file, in Visidata everything is a sheet. Once you learn how to use something about sheets in one context, you can use it everywhere else in Visidata. The selection mechanism allows you to co mbine multiple ways to select rows of data from manually picking to using regexes to Python expressions. And then you can dive into only that selected subset of data. Both of these are very powerful features.

[anjakefala]

One of mine!

I needed to find which of the hdf5 files were truncated, and therefore causing the program to crash. i had one that i suspected, so i tried opening with vd, and it verified that it indeed was the truncated file!
I could have done this with python, but it would have taken looking things up, which would have made the entire thing take longer. I just wanted a quick check.
It was nice to be able to just point vd at a file, and launch really fast. I love how I can use the same tool to look at sqlite and hdf5 files on our systems.

[anjakefala]

The most amazing part of this tool is that I'm able to open, browse, edit, and overall have functionality within a .xlsx file. As a data scientist that is fairly proficient in R, I can easily load .csv files as dataframes but .xlsx files are kind of a pain with their spreadsheet format. Also, there are all sorts of painful packages to install. All this takes is to install the tool itself (install on arch linux with yay) and the openpyxl package through pip. Based on first impressions, it looks good and feels good.

-- https://www.reddit.com/r/vim/comments/kf4wgb/visidata_a_cool_vimlike_tool_for_examining_and/gg7ism0/?utm_source=reddit&utm_medium=web2x&context=3

[daviewales]

As a Vim lover and Excel hater, I'm really loving VisiData. Previously I was using a combination of Tad + Vim + Rainbow CSV to inspect and manipulate CSV files, dropping into pandas when I needed more power. I've only been using VisiData for a week or two now, but it's already completely replaced these for quick inspections and summaries. It just feels so fast, and it's amazingly satisfying to hit Shift-F, see the histogram, select the values you're interested in with s, hit g-Enter to open a filtered table, and so on. Love it. (Selecting with |-regex is great too.)

Update:
I wanted to find out how many lettuces we eat each month. I use grocy to track our household inventory. Grocy uses an SQLite database to store the data. Before visidata, I wouldn't have even thought to inspect the grocy.db. But now I know that we eat around 4 lettuces per month... Just searched for /Lettuce in the products table, grabbed the id (68), then filtered the stock_journal table for product id 68. Then dropped all the rows where the stock was decreased (|-1gd), added a months table (="{}-{}-1".format(purchase_date.year, purchase_date.month) (old version of visidata and Python on the server...), then Shift-F to see the breakdown by month!

[anjakefala]

I have been performing bandwidth tests for my workplace using iperf. These bandwidth tests are important for learning how reliable our new connection is.

Iperf outputs its results in a log format. For the values to be informative, we need them in gigabits per second. The output was in Kilobytes per second. I also had multiple starts and stops that interspersed within the log itself.

I essentially had 10 minutes before a meeting to get a dataset ready.

I opened it up as a txt file and used search-col to find the interspersed starts and stops, and cleaned them up with delete-row. I then re-opened it as a tsv.

The values were all in one column, so I split them with : and double space. I used > to find the weird splits (that happened bc of weird formatting choices of the original data), and fix them using zd and zp. All of the units were repeated in each row, instead of in the column names, so I renamed the columns with ^, and then processed them out of the rows by using the currency type $. I froze those columns so I could then work with them.

I used = to calculate the gigabits per second, from the KiloBytes per second. I then performed a frequency table, and voila! Clean results that clearly showed how our performance test went. Returning to the dataset, it was showing me time ranges in seconds, that were a minute long, so I used addcol-incr to have a nice minute increment column.

I sent the cleaned up table to my colleague, and he could clearly see all sorts of interesting things. =)

It was really nice to be able to clean up data while being able to visually introspect. I am really appreciating all of the work underwent with Undo, because it made me feel comfortable to take my time and explore. I think the currency type is a really under-appreciated tool for dirty float clean-up, and I would like to document that. Thanks to this issue for the inspiration.

[saulpw]

From HackerNews user 'baltimore':

This is a game changer tool for exploring and manipulating tabular data. Give yourself 30 minutes to get the feel of how its wacky keybindings work (they are well thought out) and you'll be richly rewarded.

This takes its rightful place in the Hall of Very Good Tools alongside vim, htop, and multitail.

[saulpw]

From Phillip on HackerNews:

My main use case for this is: I get some random data file thrown at me. Could be json, csv, xlsx anything that has data that can somehow be interpreted somewhat tabular and I want to find out quickly what's going on.

You can really go from not knowing what's in a file to a decent understanding in less than a minute.

[anjakefala]

VisiData is proof of how powerful yet intuitive TUI tools can be. vd particularly excels when you need to answer a quick question about a dataset and avoid opening up a clunky Jupyter notebook. I particularly love the support for writing quick Python expressions to manipulate the data inline too.

Saul, thank you for bringing such a wonderful tool into the world! It really fills me with joy every time I use it. I’m quite bullish on TUI tools largely because of vd—it feels like there’s a lot more good work to be done in this space.

https://news.ycombinator.com/item?id=29063004

[setop]

This project is amazing

Gets better and more robust after each release.

Only twenty issues still open for eight hundred closed !
I myself opened three of them and they have been promptly addressed.

Only three merge request open for three hundred closed !

I now have the habit to explore my data with VD instead of a fat app spreadsheet.
People are always amazed when I show it.

I use it on massive data (over 2GB) and it is snappy.

I also really enjoy to prepare data with a replay script and then interact with them in the TUI, or export them for further report.

Thanks you so much for the effort you invest in this project.

[ghost]

I just want to say: The diff plugin is just perfect. Simple. To the point. Shows me which cells have changed. Much better than nvim -d 1.csv 2.csv!

[aborruso]

I just want to say: The diff plugin is just perfect. Simple. To the point. Shows me which cells have changed. Much better than nvim -d 1.csv 2.csv!

Where can i find documentation for this diff plugin?

Thank you

[daviewales]

Hmm... I just read the source!

Install the plugin

1. Show plugins sheet with long-command: open-plugins
2. Move your cursor to the row for the diff plugin
3. Press a to add it to visidata

Use the plugin

The plugin sets one sheet to be the 'diff sheet'. All other sheets are coloured based on a diff against this sheet.
To set the diff sheet, open the sheet, and use the long-command: setdiff-sheet.
Then open another sheet to see the diff.

A nice way to do this is to use split-pane (Shift-Z), with one file in each pane.
Then run setdiff-sheet on the top pane, and view the changes highlighted on the second file in the bottom pane:

[aborruso]

Hmm... I just read the source!

Thank you

[jli]

what are your favourite newer command line tools? like ripgrep (grep replacement), fd (find), exa (ls), bat (cat), etc

https://twitter.com/b0rk/status/1513903221466664962

visidata (vd). Amazing tool for looking at tabular data in the terminal. Also xsv for parsing CSV files.

https://twitter.com/ssdatar/status/1513905754742222851

[anjakefala]

https://twitter.com/multimeric/status/1521513572723195905

If you're doing a lot of remote #bioinformatics (on an HPC cluster or cloud node) I can't recommend @VisiData
enough. It's a lovely terminal-based "spreadsheet" viewer for Excel/CSV/parquet etc, which you can use to poke at your tabular data without needing to open R or Python.

It even supports my bad habit of storing entire molecular structure files inside cells of my parquet files, by allowing you to open individual cells inside a terminal text editor like vim or nano (Ctrl + O in edit mode).

Look at the neat columns! And toolbars! Plus it's data type aware too, if you notice the # symbol for integer columns.

[anjakefala]

From larrydavid on #visidata:

I just discovered visidata and just wanted to say that this is the best thing ever. Thanks to Saul and all contributors for creating this powerful tool!

Currently I'm mainly using it to inspect large machine learning datasets to look for problems.
I especially like that I'm able to instantiate it from PDB and inspect pandas dataframes for problems when I'm debugging data pipelines.

This tool would be so valuable to machine learning people and data scientists. I'm surprised there's not a blog post about it on towardsdatascience.com or something like that.

[frosencrantz]

Tweeted by Visidata and Saul, this MetaFilter thread shows lots of ❤️:

• https://www.metafilter.com/195489/Has-anyone-ever-used-grep-cut-awk-sort-uniq-with-a-csv-file

[Shahin-rmz]

Visidata, is by no means complete. It is first step of new age comuter users who discovered terminal, vim and live in 21st century.
whenever I have to deal with data, I use Visidata as first tool.
It needs to be even more powerfull. But it's the right direction. Cheers to the contributers.

[anjakefala]

I'll say it again, Visidata is one my favorite new CLI/TUI tools over the last few years. It makes it extremely easy to explore/modify structured data in the terminal.

• leonim (https://news.ycombinator.com/item?id=32442010)

[saulpw]

Also leonim:

To me it is amazing at exploring data. I wish we had that same interface on every place I'm looking at a table of data.

I can't see it replacing Excel or other spreadsheets for people who make heavy use of spreadsheets in their jobs. For a terminal junkie like me, who wants/needs to make sense of data that is much bigger than my screen, Visidata is much better than a spreadsheet. I can filter and partition data easily.

I used to have to use perl/awk/sed/sort/comm/grep to help me put that data in some format that was useful. Visidata is much easier and faster to use.

[reagle]

I was thinking it be interesting to see vd face off against Excel in an analysis competition! For example, different competitors are given the same data set and their tool of choice, asked a question, and see you gets the correct answer most quickly.

Also, I think it's use in teaching would be great.

[saulpw]

I would love to see that happen. BattleSheets!

[yarikoptic]

visidata is great and I promote it everywhere I go. In my field we use lots of .tsv files and I think people do not use them enough because typically they do not know about visidata! ;)