UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 43513811 #23 141 points 65 comments -> Trump's attacks on universities get darker, with shadows reaching our shores
- 43511464 #9 168 points 65 comments -> iCloud Mail has DNS misconfigured?
- 43478480 #24 15 points 10 comments -> Window cleaner in quest to confirm priceless Shakespeare portrait
- 43482026 #9 7 points 0 comments -> I Went Undercover in Crypto's Answer to Squid Game. It Nearly Broke Me
- 43515268 #28 10 points 0 comments -> The Design of the New Swiss Passport
- 43515169 #28 12 points 2 comments -> Therapy chatbot trial yields mental health benefits
- 43515676 #28 4 points 0 comments -> Targeting of Tufts Student for Deportation Stuns Friends and Teachers
- 43514459 #8 24 points 2 comments -> Building a search engine from scratch, in Rust: part 1
- 43515869 #13 6 points 3 comments -> ChatGPT is turning everything into Studio Ghibli art
- 43475982 #30 34 points 5 comments -> The Curious Case of Beam CPU Usage (2019)
- 43479262 #10 16 points 0 comments -> AM Radio Will Never Die
- 43517352 #10 12 points 2 comments -> Huge Structures Discovered Under Pyramids?
- 43517613 #18 15 points 1 comments -> Jeremy Howard taught AI and helped invent ChatGPT. He fears he's failed
- 43517262 #22 22 points 40 comments -> AI generated Ghibli images go viral as OpenAI loosens its rules
- 43518064 #28 12 points 3 comments -> Show HN: Job Application Bot by Ollama AI
- 43518030 #16 19 points 3 comments -> Proof that Steins;Gate world is a running simulation
- 43492378 #20 4 points 0 comments -> Why do blue-footed boobies have blue feet? And 8 other blue-footed booby facts
- 43519459 #8 12 points 3 comments -> Vibe Coding with Cursor
- 43519678 #22 7 points 0 comments -> Ex-FCC chairs from both parties say CBS news distortion investigation is bogus
- 43519719 #9 9 points 1 comments -> Deep Musk
- 43520302 #30 7 points 0 comments -> Rescission of the Final Scientific Integrity Policy of the NIH
- 43521521 #11 2 points 0 comments -> Show HN: Online tool to show white fullscreen page
- 43523165 #24 6 points 0 comments -> Hegseth's Wife Tagged Along for Top Secret Military Meetings
- 43523873 #7 8 points 5 comments -> Show HN: LinkedIn's Dead. Developers Are Moving to Grids
- 43498071 #12 56 points 17 comments -> Rust Adopting Ferrocene Language Specification
- 43499319 #13 62 points 7 comments -> Charlie Brown and Franz Stigler incident (1943)
- 43525937 #2 10 points 2 comments -> The First LLM
- 43526270 #23 4 points 0 comments -> Satanist leader's attempt to hold Black Mass in Kansas Statehouse sparks chaos
- 43524385 #23 12 points 3 comments -> Lehmer's Continued Fraction Factorization Algorithm
- 43529362 #3 10 points 8 comments -> Executive order phases out U.S. Treasury paper checks
- 43528496 #19 42 points 41 comments -> Can Style Be Timeless?
- 43530751 #24 41 points 40 comments -> Fiber optic drones are changing combat operations
- 43531494 #5 16 points 5 comments -> Agentic AI Needs Its TCP/IP Moment
- 43532939 #1 11 points 5 comments -> GPT-4o draws itself as a consistent type of guy
- 43532736 #14 3 points 1 comments -> Custom AI Chatbot Development Services: Restrictions and Opportunities
- 43533634 #14 11 points 4 comments -> Net migration into the UK has tripled post-Brexit
- 43535311 #30 7 points 0 comments -> Trump's USAID Cuts Hobble Earthquake Response in Myanmar
- 43525246 #19 17 points 10 comments -> Tiniest Flying Robot (Wingspan <1cm) Soars Thanks to Magnets
- 43537580 #6 -> ICE deporting legal Permanent Residents, Refugees, and immigrants
- 43537624 #20 2 points 0 comments -> Chomsky: A "Study in Total Depravity"
- 43498465 #20 5 points 2 comments -> A Man Who Went to Fake Prison Also Went to Real Jail
- 43537808 #16 71 points 55 comments -> The Average College Student Is Illiterate
- 43541428 #4 5 points 1 comments -> Addressing Burnout – Libera Chat
- 43541926 #26 4 points 1 comments -> Tips for Travelers Entering the U.S. Now: Check Your Visa, Turn Off Your Phone
- 43542151 #21 -> The Demand Side of Democratic Backsliding
- 43543492 #5 5 points 0 comments -> ØNTHR The World’s First Perfect Guaranteed $0.00 Stablecoin
- 43543881 #24 1 points 0 comments -> Goodbye Offpunk, Welcome XKCDpunk
- 43517243 #19 7 points 0 comments -> Open-source browser-use/CUA equivalent for androids: mobile-use
- 43545674 #10 2 points 0 comments -> How Airbnb Measures Listing Lifetime Value
- 43546647 #29 3 points 0 comments -> Torrent Site Uploader and Member of 'The Scene' Sentenced to Prison in Denmark
- 43546669 #7 -> The Turd Reich [pdf]
- 43548454 #10 5 points 1 comments -> Show HN: Switch 5 less apps a day using this iPhone app
- 43516824 #15 5 points 0 comments -> Exploring Anagram Search in PostgreSQL with Recursive CTEs
- 43520005 #26 24 points 40 comments -> Chess324 – A Chess Variant to Reduce Draws
- 43533104 #20 8 points 2 comments -> The March of the Vegetables Parade
- 43552884 #15 -> Show HN: Fuck Lightroom and Fuck Adobe: How Adobe Is Systematically Predatory
- 43553134 #17 16 points 3 comments -> After Crash FAA Change Requires All Aircraft at Reagan to Broadcast Positions
- 43553271 #30 5 points 1 comments -> Call for testing: OpenSSH 10.0 (DSA support removed)
- 43554064 #10 25 points 40 comments -> Travel Guide for Stateless People
- 43555724 #4 63 points 15 comments -> An 'administrative error' sent a Maryland man to an El Salvador prison
- 43555814 #2 32 points 37 comments -> Don't Bother with Vibe Coding
- 43556281 #9 11 points 2 comments -> Mike Waltz Used Personal Gmail for Government Communications: Report
- 43556463 #12 42 points 9 comments -> Tesla (TSLA) announces 336,681 deliveries, far worse than expected
- 43556443 #5 41 points 11 comments -> Tesla suffers worst quarter since 2022 as deliveries tumble
- 43556355 #12 9 points 4 comments -> What I would do if I was 18 now
- 43556344 #5 7 points 4 comments -> Starlink sets up subsidiary in Faroe Islands
- 43556624 #16 8 points 3 comments -> American shakedown: Police won't charge you, but they'll grab your money
- 43556841 #18 3 points 0 comments -> Anyone who trusts an AI therapist needs their head examined
- 43557655 #23 3 points 1 comments -> Global scam industry evolving at 'unprecedented scale' despite recent crackdown
- 43558242 #13 12 points 2 comments -> Amazon Said to Make a Bid to Buy TikTok in the U.S.
- 43558095 #29 5 points 0 comments -> We Need to Rename ADHD (2021)
- 43531816 #29 32 points 16 comments -> You're protecting your data wrong – Introducing the Protected Query Pattern
- 43558777 #23 5 points 1 comments -> PayPal launches remote and local MCP servers
- 43559265 #28 3 points 0 comments -> Waymo's Foundation Model for Autonomous Driving with Drago Anguelov [video]
- 43558352 #16 34 points 21 comments -> Bare: Run JavaScript Everywhere
- 43560751 #9 14 points 22 comments -> It's Unreal Just How Awful 'Real ID' Is (2020)
- 43562005 #24 38 points 25 comments -> Wikipedia, says its bandwidth costs have gone up 50% since Jan 2024
- 43563608 #28 13 points 4 comments -> Australian beef singled out in latest USA tariffs
- 43563389 #27 6 points 3 comments -> Show HN: OpenTariffs – Tariff Percentages by Country
- 43527933 #23 8 points 0 comments -> The Carbon Bankroll 2.0 [pdf] (2024)
- 43569636 #19 10 points 0 comments -> Decapitation Strike
- 43570859 #7 5 points 0 comments -> I Hired 5 People to Sit Behind Me and Make Me Productive for a Month
- 43573774 #19 12 points 1 comments -> US tariff protection decimated American shipbuilding; could do same to cleanTech
