Improving Nokogiri documentation

[BurdetteLamar]

I'm thinking to spend time this year with some of the doc here. I see:

• Formats: HTML4, HTML5, XML.
• Activities: parsing, searching, editing, generating.

Any thoughts about what most people are doing? And what could best use the work?

[flavorjones]

Hi @BurdetteLamar! Thank you again for being willing to spend your time helping improve Nokogiri. ❤️

I took a look at some of the issues I wrote up at sparklemotion/nokogiri.org/issues and some of those are still valid and valuable, and other maybe not so much ... I think the ones that might be most valuable -- and what's missing from Nokogiri docs today -- are short how-to documents.

I was recently introduced to https://diataxis.fr/ which describes some mental models for characterizing and improving technical documentation. This 2x2 in particular helped me think about Nokogiri:

Breaking it down:

• Tutorials (learning-oriented experiences) The nokogiri.org site today has a few of these which broadly introduce some concepts.
• Explanation (understanding-oriented discussion) I've started collecting some deeper "why" docs in the /adr subdirectory and in code comments.
• Reference (information-oriented technical description) This is obviously the rdoc strings around classes and methods, which are not superb today but have steadily been improved over the last few years.
• How-to Guides (goal-oriented directions) We basically have none of these outside of a few short examples in rdoc strings.

There is a great book called The Ruby Cookbook which hasn't been revised in nearly a decade, and it has a chapter on XML and HTML but barely mentions Nokogiri (because Nokogiri had not yet been released when the first edition came out in 2006). It's always been an idea in the back of my head to try to rewrite that chapter using Nokogiri instead of REXML.

Anyway, that's what's in my head. Is that helpful at all?

[BurdetteLamar]

A cookbook seems like a good idea. I've done Ruby recipes before, over at CSV. (Those are very old, and may not be my best work.)

Those in the Ruby Cookbook, 2nd Edition may not be the best choices for us (see list below). Suggest we make a list of those we think most important (new Issue?), and maybe gather suggestions from others as well.

In the Cookbook:

12.1 Checking That XML Is Well Formed
12.2 Extracting Data from a Document’s Tree Structure
12.3 Extracting Data While Parsing a Document
12.4 Navigating a Document with XPath
12.5 Converting an XML Document into a Hash
12.6 Validating an XML Document
12.7 Substituting XML Entities
12.8 Creating and Modifying XML Documents
12.9 Compressing Whitespace in an XML Document
12.10 Guessing a Document’s Encoding
12.11 Converting from One Encoding to Another
12.12 Extracting All the URLs from an HTML Document
12.13 Transforming Plain Text to HTML
12.14 Converting HTML Documents from the Web into Text
12.15 Creating a Simple Feed Aggregator

[flavorjones]

👍 I've posted on bluesky and mastodon asking people to comment here if they want/need a recipe for a Nokogiri task.

[BurdetteLamar]

May I also ask on dev.to? (Probably best not on reddit.)

[flavorjones]

Please do! 👍

[BurdetteLamar]

https://dev.to/burdettelamar/nokogiri-cookbook-4p1d

[jeremysmithco]

Hey there! Just found this by way of Short Ruby. I'd be very excited to see expanded/improved Nokogiri documentation!

This may or may not be a good fit for what you are thinking, but for one project I work on, we parse and analyze a lot of scraped web pages using Nokogiri. Common tasks for a given page:

• Build a content outline based on the heading tags from a page, knowing that e.g. sometimes h1 is skipped or otherwise isn't the page title
• Extract and store all images from a page, skipping things like header images and navigation icons, but including base64-encoded image data
• Extract and clean the primary content of the page (i.e. article, blog post) skipping header, footer, navigation, stripping ad content, etc. (we've used https://github.com/tammersaleh/ruby-readability for this, but it's not perfect)

Part of the challenge is learning how to use Nokogiri well, but another part is just figuring out how to deal with all the weird things you see in the wild. And that part may be difficult to cover...though I would say it might be the most helpful, because often the recipes I find assume clean/stable markup (e.g. "to find the price on the product page, just extract the text from the .price class").

Another challenge I've had is learning XPath. Again, this isn't strictly a Nokogiri thing, but I've just not found that many good resources for learning it. I've been surprised by some of the things I could do, for example:

# Remove all SVG images
body.xpath(".//img[substring(@src, string-length(@src) - 3) = '.svg']").remove

# Remove all element attributes except for src and href
body.xpath(".//@*[not(local-name()='src' or local-name()='href')]").remove

Hope something there is helpful! Let me know if you have questions. Thanks!

[BurdetteLamar]

I had hoped that this would be my "year of Nokogiri doc"; we are seven weeks in and I have only three merged PRs and two pending. There won't be many (if any) other such years, as I've just turned 82.

I need timely reviews for the doc PRs I put up. I've found (over at ruby/ruby and other ruby/*) that each reviewed PR informs me about what to do next, and, especially, how. Therefore I've learned not to get out over my skis by writing too much at at a time. Which, in turn, means that I should have a very small number of small open PRs at any one time.

@flavorjones, I see from the traffic that you are busy. Are there other potential reviewers?

• Owners listed over at RubyGems,org: @tenderlove and @knu?
• Heavy users, such as @jeremysmithco?
• Well-qualified volunteers recruited at the ruby-lang Slack?

[flavorjones]

I will review today. I apologize for not being timely, and will try to do better. It's been challenging balancing work, life, and OSS (which is a hobby).

[BurdetteLamar]

I will review today. I apologize for not being timely, and will try to do better. It's been challenging balancing work, life, and OSS (which is a hobby).

Please @flavorjones, do no apologize. Like others in its user community, I'm grateful to have Nokogiri and grateful to you and the others who've work on it.

[jeremysmithco]

as I've just turned 82

Whoa ok, life goals! 😊💪 Hope I'm pushing out PRs at 82!! 👏

BTW, I'm probably more of a moderate Nokogiri user. I was reading over the discussion on #3431 and I wouldn't have been able to weigh in meaningfully on something like that, for example. But if you need a second pair of eyes on a code examples, particularly around parsing existing markup, feel free to ping me!

[BurdetteLamar]

Better than pushing up daisies!