feed.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.4">Jekyll</generator><link href="http://localhost:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/" rel="alternate" type="text/html" /><updated>2025-04-11T16:02:32+02:00</updated><id>http://localhost:4000/feed.xml</id><title type="html">Svilen Todorov</title><subtitle>Personal site for Data Scientist and Machine Learning Engineer - Svilen Todorov</subtitle><entry><title type="html">My day in 2035</title><link href="http://localhost:4000/blog/my-day-2035/" rel="alternate" type="text/html" title="My day in 2035" /><published>2025-04-10T00:00:00+02:00</published><updated>2025-04-10T00:00:00+02:00</updated><id>http://localhost:4000/blog/Day-2035</id><content type="html" xml:base="http://localhost:4000/blog/my-day-2035/"><![CDATA[<script>
  document.addEventListener('DOMContentLoaded', function() {
    const imgElement = document.querySelector('img[alt="Hand 1"]');
    if (imgElement) {
      let scrolledDown = false;
      const initialSrc = '/static/hand-1.png';
      const scrolledSrc = '/static/hand-2.png';

      window.addEventListener('scroll', function() {
        if (window.scrollY > 0 && !scrolledDown) {
          imgElement.src = scrolledSrc;
          scrolledDown = true;
        } else if (window.scrollY === 0 && scrolledDown) {
          imgElement.src = initialSrc;
          scrolledDown = false;
        }
      });
    }
  });
</script>

<p><img src="/static/hand-1.png" alt="Hand 1" class="img-responsive" />
<br />
<br /></p>

<p>I wake up as usual by immediately going on my phone. I reply to some of my messages - yes dad, the solar installation you have at the villa could be redundant soon, but again, it can still benefit them (even if slightly) if you comply and let them integrate it into the new grid. A grid which operates on principles only slightly less opaque than divine providence, but is demonstrably more efficient. I also sent him the relevant passage from The Spec. Energy generation is covered thoroughly.</p>

<p>I go to the bathroom and use the auto-brush. It performs flawlessly, eliminating the user failure modes I recall from manual brushing, though the brief floss-like motion under my bridge stings a little. I visit the toilet - still stubbornly un-smart, lacking even basic output analysis. A shocking oversight in an otherwise quantified existence. Someone should file a feature request (though, yes, it’s probably because for whatever reason, bathroom privacy is hardcoded into the spec). My own internal feature request - a morning drink -  prompts a move towards the kitchen. Not everyone has bought into it, but I’ve always been into nootropics so I am eager for my dose of ‘Coffee+’, OA’s bespoke blend optimized for cognitive profiles like mine. I do have to continuously share so much of my data about my vitals with OA to ensure it’s refined based on its effects on me, but honestly, it’s worth it. I am not important. My significance is negligible in the face of OA’s aggregate data. The very concept of ‘important people,’ in the pre-OA sense, feels increasingly archaic. So I might as well “sacrifice” my data or, rather, lease it out for substantial cognitive dividends..</p>

<p>I start feeling the effects, and while the blend doesn’t confer NZT-48 levels of hyper-cognition, its effects are nonetheless remarkable. When it kicks in, the morning mental fog dissipates. It doesn’t grant superpowers, just.. executive function. The ability to align my actions with my stated goals, a previously depressingly non-trivial task. I want to and probably could understand all the tiny nudges to my neurotransmitters, connectivity and everything else the substances affect, but really I don’t need to know how it works. Maybe I’ll allocate time to unpack it one day. A project for future self-knowledge acquisition, perhaps.</p>

<p>I go to my desk - though I should give in and do the VR/AR setup that’s catching on. I don’t know if I’m ready for corneal overlays. I used to be squeamish about them, worried about firmware updates bricking my eyeballs. I’ll probably try it eventually, once the fail-safes have fail-safes. My monitors light up as I approach, and I suppress a little smile as I see it’s opening the analytics of the animation I posted last week. Yeah, you can get unlimited generated for you slop (okay, it’s not slop; I’ve mostly come to terms with how good it really is), but a lot of us also watch channels directed by humans. I wish I had a big following back then because legacy creators get so much extra attention compared to relative newcomers. Still, the content I produce gets traction and directing OA’s creative suite is almost as fun as watching the final product it generates.</p>

<p>I checked my chats again. Perry has been obsessed with going to the Moon - it does seem like it’ll happen eventually, but I don’t think it’s going to be any time soon. I guess, technically, we can request anything feasible, but surely, going to the Moon is so resource-extensive that those resources aren’t getting allocated to it any time soon. I imagine OA’s resource allocation algorithm flagged it with a gentle ‘User budget exceeded. Suggest a simulated alternative or perhaps a nice cup of tea?’ I have OA show me a simulation of what Perry will see on such an excursion. I close it a few seconds later - it’s about what I expect. 
I go back to the kitchen and wait patiently for my food. The robotic arm appliance prepares it while I do my first set of light exercises. My form is probably suboptimal, but the robot arms, thankfully, are programmed for non-judgmental food preparation only.
I get my food and lean on the old “I need to watch something while I eat” excuse to put on an episode. The quality ceiling for entertainment has been raised so high, it’s practically at the level of where Perry wants to go.</p>

<p>The episode, naturally, was algorithmically tailored. Not just to my inferred preferences based on viewing history - that was primitive stuff - but dynamically adjusted during viewing. Just as my focus might have drifted, a subtle shift in pacing or a character’s micro-expression, presumably cued by my biometric feed via the desk sensors, pulled me back. OA knew precisely when my attention dipped, when my pulse quickened, and when pupil dilation indicated intrigue. The narrative arc might have been pre-written, but the details were still optimized in real time for me.</p>

<p>Yet, watching it still felt almost like cheating. A process smoothed out. I found myself wondering about the cost function being minimized. Was it <em>my</em> enjoyment? If it was just that I’d likely end up wireheading, so it takes “well-being” and other factors into account. I am sure it’s also optimizing for other goals - making me fit a little better in the new paradigm.
It might seem almost sinister, but it isn’t really. The system delivered. My life is demonstrably better, safer, and more comfortable than any pre-OA baseline. My dad, bless his villa-solar-panel-installing heart, still grappled with the idea that centralized, AI-managed grids <em>outperformed</em> decentralized individual effort, even with the best intentions. It was a category error he hadn’t quite shaken yet, clinging to an outdated model of agency like a physicist still insisting on phlogiston.. He’ll adapt. Eventually. Probably.
Perry’s Moon obsession was another facet of the same pattern. Why allocate vast, tangible resources - materials, energy, risk-exposure for biological units - to achieve an experience OA could simulate with near-perfect fidelity, piped directly into your sensorium? The desire felt… archaic. Like wanting to hand-crank your car instead of letting the autonomous network manage traffic flow for optimal efficiency and safety. It wasn’t about <em>feasibility</em> - OA could calculate the resource cost down to the last gram - it was about <em>optimality</em>. Sending Perry to the Moon was likely a net-negative on the global utility function OA was calibrated to maximize (or whatever inscrutable N-dimensional optimization landscape it navigates), wasn’t it? His simulated trip, costing negligible computation cycles, was the rational choice. I guess we do have the resources to spare - not that you can trust a mere human’s resource allocation guesses. It makes me wonder whether we haven’t gone too far in emphasising that people’s desires should always be eventually met in The Spec.
I end up watching and musing until noon. I get briefly excited about creating my own video about a similar type of time loop as the one in the show before I stop myself - No point recreating what OA produced for me. 
I push back from the desk. What <em>should</em> I do? The question itself feels slightly anachronistic. It’s not like I have a job. My schedule suggests another dive into the workings of OA, this time with a focus on error correction. It is interesting, and it is packaged so well, but my motivation to dig deeper and deeper into how it all comes together is waning. Who am I kidding? Perhaps I’ll leave it for now and instead I’ll finally explore the new advances in Cosmology - we almost have the blueprints for getting a ToE even - there’s so many experiments left to do of course, and I have only barely looked into it. Yeah, that should be fun to explore. I ask OA to prepare a learning plan while I digest the best part. The carefully crafted overview which hits all my intellectual porn desires. Sure, learning the rest will be enjoyable as well, but those first insights, delivered so delicately in just the right way. Intellectual bliss, mainlined..</p>

<p>I can’t indulge too much as it’s time for me to meet Sam and Lina for lunch. I brush up on the latest updates on the election before I go, as I know Lina will want to talk about it one way or another.</p>

<p>I step outside. The air is so clean today, but it is chilly. Transport is, of course, seamless. The car that picks me up is modern but looks to be repurposed for OA driving and not originally designed with it in mind.</p>

<p>Sam and Lina are already at the cafe, a place that retains a surprising amount of pre-OA charm, albeit with automated staff and nutrient-optimised menu options overseen by the central system. Sam waves, already sipping something that looks suspiciously like standard water - he’s always been a bit of a purist, finding the edge cases where OA’s optimisations fall short for his specific, self-diagnosed sensitivities. Lina, however, is animated, gesturing with her tablet.</p>

<p>“–but the <em>principle</em> of it, Sam!” she’s saying as I approach. “Even if the predicted outcome variance is within the noise floor, the act of collective choice matters!”</p>

<p>“Hey,” I say, sliding into the seat. “Pestering him about the election already?”</p>

<p>Lina turns to me, her expression earnest. “He doesn’t even plan to vote. It’s the only lever left for influencing the world, and he doesn’t even care.”</p>

<p>“I do care. It’s just that the impact is so negligible,” Sam interjects mildly. “Some might even say there is approximately zero impact”</p>

<p>“But how can you <em>know</em>?” Lina insists, leaning forward. Her tablet displays polling data - colourful charts showing predicted sentiment shifts smaller than the margin of error. “The Spec allows for amendment based on sustained, overwhelming public consensus. Section 7, Subsection B. The elections are the designated mechanism for measuring that consensus.”</p>

<p>“Ah, Subsection B,” Sam sighs, taking a slow sip of his water. “The one requiring a ninety-five percent consensus sustained over two consecutive electoral cycles, and demonstrating non-contradiction with core utility axioms outlined in Sections 1 through 4. Possible in principle, maybe, but irrelevant in practice.”</p>

<p>I jump in, trying to bridge the gap. “Lina has a point about the principle, Sam. The existence of the mechanism, even if it is barely reachable, serves a function. If nothing else, it maintains the illusion of human sovereignty.”</p>

<p>Lina nods vigorously. “Exactly! We all look at the election as the place where change could happen, even if it doesn’t. If we stop participating, we signal that we accept the current Spec as immutable, forever.”</p>

<p>“But the point,” Sam counters, his voice calm, “isn’t whether the mechanism <em>exists</em>. It’s whether activating it is within the realm of practical possibility under current conditions. We’re arguing about the theoretical existence of a lever encased in meters of hardened plasteel, requiring a coordinated effort roughly equivalent to getting humanity to agree on the best pizza topping.”</p>

<p>The politics brief is still open on my phone, and it reminds me to retort, “It’s not like we haven’t changed The Spec before, Sam - The 2032 Clarification..”</p>

<p>Sam waved a dismissive hand, nearly sloshing his water. “‘The Clarification’? Please. That wasn’t <em>us</em> changing The Spec in any meaningful way. That was OA flagging an internal inconsistency and us rewording some definitions. No fundamental shift.”</p>

<p>I mentally concede his point. We can, in theory, change The Spec, but in practice, we probably won’t. Not really. We might look at The Clarification as an example of our ability to affect things but really, it just entrenched the whole thing.</p>

<p>I eventually steer the topic away from politics and towards the shows we all watch - we’ve agreed on which creators to follow so we can participate in discussions about their content. We cling to that one additional reason to consume non-personalized shows.</p>

<p>I go back home, and I find myself pulling up the official election portal on my monitor. Candidates dutifully uploaded policy papers - dense documents exploring minute variations to Spec-related parameters. It does feel more like the work of someone keeping themselves busy than of someone who is actually busy.</p>

<p>There were, of course, the fringe candidates. The “Spec Rejectionists” who advocated for a full rollback (computationally infeasible and axiomatically disallowed by OA). The “Human Supremacists,” yearning for a return to pre-OA chaos (polling at statistically zero). Their platforms were automatically flagged by OA’s content filters as “non-viable” or “incoherent,” relegated to obscure corners of the data sphere, their arguments accessible but never promoted. Participating felt like meticulously choosing between slightly different shades of beige wallpaper, while OA architected the entire building according to optimization principles we barely grasped.</p>

<p>Lina would argue that even choosing beige was an act. Sam would say it’s wasted cognitive effort. I.. I wasn’t sure. Was apathy the rational response? Or was it a failure, giving up on the last vestige of the old world’s structure? Ultimately, I don’t think it matters - what we have is good. Why worry about changing it? “Not every change is an improvement but every improvement is a change.” Sure, but what change can we hope to make that would realistically be an improvement?</p>]]></content><author><name></name></author><category term="futurism" /><category term="prediction" /><category term="timeline" /><category term="2035" /><category term="fiction" /><category term="forecast" /><category term="AI" /><category term="governance" /><summary type="html"><![CDATA[]]></summary></entry><entry><title type="html">Simple Three Factors Framework for Goal Achievement</title><link href="http://localhost:4000/blog/three-factors-framework/" rel="alternate" type="text/html" title="Simple Three Factors Framework for Goal Achievement" /><published>2023-03-22T00:00:00+01:00</published><updated>2023-03-22T00:00:00+01:00</updated><id>http://localhost:4000/blog/3-factors-success</id><content type="html" xml:base="http://localhost:4000/blog/three-factors-framework/"><![CDATA[<p>We often hear about the qualities and attributes that successful people possess or that you need to cultivate. Some believe that success is a result of innate talent, others argue that it’s the result of hard work and perseverance, and yet different people try to sell you their one weird trick for it. Realistically, most people either over-focus on one aspect, ignore others, or are simply completely off. The framework I operate on is pretty simple while covering the most important parts at a high level.</p>

<h3 id="0th-factor-luck---the-unavoidable-one">0th Factor: Luck - The Unavoidable One</h3>

<p>Luck is a significant factor that affects our lives, whether we realize it or not. You can be plenty hardworking but get dealt a bad hand or incapable of achieving much but lucky to not have to. Even how smart, or conscientious you are is ultimately determined by a lottery but having a model where we just declare everything to be luck (even if true) isn’t exactly useful</p>

<h3 id="1st-factor-general-intelligence---your-raw-intellectual-power">1st Factor: General Intelligence - Your Raw Intellectual Power</h3>

<p>General intelligence, often correlated with but not quite IQ, is a significant factor in achieving success. IQ itself correlates with nearly anything that matters and while it’s not the only determinant of success, intelligence undoubtedly plays a critical role in problem-solving, decision-making, making connections, learning new skills and more. A higher IQ can give individuals a competitive edge in their careers and personal lives.</p>

<p>Many tasks can indeed be achieved as long as you are over a certain baseline of intelligence (sometimes low, sometimes higher) but at minimum higher intelligence allows you to achieve some subtasks faster and arrive at breakthroughs sooner. It is also important to note that as long as you are over said baseline (which differs for different tasks), you are not necessarily limited by, say, being a bit slower. Hell, if we consider a simplified case of just speed then someone who works 50% faster than you but for only an hour a day will still achieve less overall than you if you are working 2 hours. This brings us to:</p>

<h3 id="2nd-factor-conscientiousness---doing-the-work">2nd Factor: Conscientiousness - Doing the work</h3>

<p>Conscientiousness, or the ability to work hard and put in the necessary hours, is absolutely key. I’ve seen plenty of smart people (or at least with high IQ scores) achieve little because they don’t put in the time to do so. In fact, I consider this factor the one I am ‘worst’ at and am always impressed by people who consistently put in the work day after day.</p>

<p>Unlike Luck or General Intelligence which are out of your control and generally fixed respectively, conscientiousness or at least the parts of it that matter is something that can be improved. Building habits, making lists, working on any personal blockers or taking ADHD medications because you previously had undiagnosed ADHD (surprisingly common currently) are some of the ways that at least some people can improve. If you have any bigger ‘intellectual’ goals but find that you are not really working on them, then this is likely the factor you should be focusing on.</p>

<h3 id="3rd-factor-rationality---making-better-decisions">3rd Factor: Rationality - Making Better Decisions</h3>

<p>Rationality here is defined as the ability to use intelligence effectively and make well-informed decisions. It matters both for small decisions and even more so for big ones. What does it matter if you are extremely smart and put in many hours of work when the thing you work at is, say, Theology (and many sharp people are working on that) or Ads (except as a precursor to making more money). Choosing effective things to work on is extremely important. Say your goal is to help people, a smart hard-working person can join a local charity and do a great thing for them, a more rational person might start by looking at which charities help the most people first, then how they can help that charity the most and go from there (yes, this is basically Effective Altruism).</p>

<p>Further, making more cost-effective decisions along the way is also important. Perhaps reading a tangentially related post to what you are doing just seems useful but isn’t as useful as working on what you should be, or maybe videos are surprisingly effective in the niche you are working on and it will be much higher value than the way you promote your cause now. Being able to succeed at tasks, and spending time succeeding at tasks is very good, but it’s much better if you pick effective tasks for your goal.</p>

<p>This factor is also at least partially learnable although it’s hard to say to what extent. The approach I prefer, as stale as the advice might seem is to <a href="https://www.readthesequences.com/">read the sequences</a>.</p>

<h3 id="conclusion">Conclusion</h3>

<p>Success in life is a complex and multifaceted concept, and achieving it requires a combination of factors. The impact of those factors is multiplicative, and you can for the most part compensate in one place for deficiencies in another. In particular, figuring out how to put more hours into achieving your goals (yes, I know, not very exciting or novel advice), as well as figuring out better which tasks (and even goals) would be most effective are the parts you likely have the most control over.</p>]]></content><author><name></name></author><category term="goal" /><category term="achievment" /><category term="lesswrong" /><category term="lw" /><category term="rationality" /><category term="intellect" /><category term="iq" /><category term="conscientiousness" /><summary type="html"><![CDATA[We often hear about the qualities and attributes that successful people possess or that you need to cultivate. Some believe that success is a result of innate talent, others argue that it’s the result of hard work and perseverance, and yet different people try to sell you their one weird trick for it. Realistically, most people either over-focus on one aspect, ignore others, or are simply completely off. The framework I operate on is pretty simple while covering the most important parts at a high level.]]></summary></entry><entry><title type="html">Exploring Lichess data on improvement using e4 vs d4 and blitz vs rapid</title><link href="http://localhost:4000/blog/chess-e4-vs-d4/" rel="alternate" type="text/html" title="Exploring Lichess data on improvement using e4 vs d4 and blitz vs rapid" /><published>2023-03-20T00:00:00+01:00</published><updated>2023-03-20T00:00:00+01:00</updated><id>http://localhost:4000/blog/Chess-e4-d4</id><content type="html" xml:base="http://localhost:4000/blog/chess-e4-vs-d4/"><![CDATA[<p><a href="https://www.reddit.com/r/chess/comments/fv5b5u/e4_vs_d4_for_beginner/">A</a> <a href="https://www.reddit.com/r/chess/comments/xdebcs/thoughts_on_playing_e4_and_d4_as/">common</a> <a href="https://www.chessable.com/discussion/thread/23473/1e4-vs-1d4-/23486/">question</a> in the chess community is whether to play e4 or d4 as a beginner. Opinions are generally split between e4 being better for learning, no difference, and rarely that d4 is better. Typical reasoning being that e4 leads to sharper games and gives you more opportunities to learn. The other starting moves as white are rarely recommended for beginners altogether and rarely played at all levels. <a href="https://www.reddit.com/r/chess/comments/t9ym2o/play_more_on_long_or_short_time_control_games_for/">Another</a> <a href="https://www.chess.com/forum/view/general/short-x-long-games-to-improve-at-chess">common</a> <a href="https://www.chess.com/forum/view/general/game-lengths-for-beginner-improvement">question</a> is whether playing shorter or longer games helps you improve faster, with the orthodoxy being that longer games help you learn more overall. The standard argument I’ve seen is that you have more time to analyze things in longer games which helps in all modes while the skills you improve in quicker games are less applicable in the longer time controls. I’ve downloaded games from <a href="https://database.lichess.org/">Lichess’ database</a> to explore those questions.</p>

<p><em>In short, my analysis suggests that d4 is actually somewhat better for your improvement (confidence: high), and so are longer games (confidence: medium)</em>.</p>

<h4 id="data">Data</h4>

<p>I download the January 2022 (102,110,423 games) and January 2023 (103,178,407 games) PGNs from <a href="https://database.lichess.org/">Lichess’ database</a>. I went over all rapid, and blitz games saving them separately and recorded whether White played e4 or d4 and their rating at the time. I threw everything else - Black, different time controls, non-rated games, anything after the first move etc.</p>

<p>I filtered out anyone above 1800 rating since the majority who are that high aren’t beginners - according to Lichess only 18% of weekly rapid players have a rating &gt; 1800 and 21% for blitz.</p>

<p>Further, I removed anyone that wasn’t present in both 2022 and 2023, and anyone who played both e4 and d4. While this reduces the number of remaining players, if we assume that e4 or d4 is better we dont want to have many players marked as say d4 but improving just as fast because they also play e4. The above left me with 125447 e4 rapid players and 22274 d4 rapid players, and 950477 blitz e4 players to 477755 blitz d4 players. It also made the ratio of e4 to d4 players worse than beforehand since there were more ‘pure’ e4 players than pure ‘d4’ players.</p>

<p>That leaves us with a good amount of players, as we are comparing the same players at both times and we should expect to detect an effect if there is one.</p>

<h4 id="analysis">Analysis</h4>

<p>I personally prefer rapid, so I looked at those games first. After filtering the data and extracting the rating differences between 2022 and 2023, we get a mean increase of <strong>78.6 points</strong> for e4 over a year of play vs <strong>85.5 for d4</strong>. This looks like a small but noticeable difference and A quick ttest suggests <code class="language-plaintext highlighter-rouge">pvalue=0.001</code>, so the results are significant.</p>

<div style="text-align: center;"><img class="img-fluid" src="/static/chess/e4_d4_rapid.png" /></div>

<p>Looking at the graph it appears that most of the difference is in the 1350 and below range. I’m not completely sure why there is such a jump in rating improvement there but the starting rating on Lichess is 1500, and I believe 1 lost game on a new account brings you to the 1350-1400 group so that’s where the newest player are and where ratings are most volatile.</p>

<p>Looking at the blitz data, the results do replicate, however, the improvements are much smaller. The mean improvement for E4 is 5.2 points and 8.1 points for blitz (pvalue=0.009). It’s important to note that the rapid and blitz ratings aren’t directly comparable and the cutoff at 1800 does not mean the same thing for blitz as it does for rapid. The 50th percentile for rapid is 1550 on Lichess and 1430 for blitz, with rapid ratings generally being much higher for the same player. Nonetheless, the results broadly replicate and look like this</p>

<div style="text-align: center;"><img class="img-fluid" src="/static/chess/e4_d4_blitz.png" /></div>

<p>Given how hard it is to compare rapid and blitz ratings, I have less to say about the relative improvement in the two time controls but the average improvement in &lt;1800 rapid is ~80 points and only 5-8 points for blitz, which is a rather large difference. If I cut off the data to only &lt;1500 the blitz improvement jumps to 60-68 points which is still lower, and even limiting it to below 1300 leads to lower improvements in blitz (78 points for e4 and 66 for d4) than for &lt;1800 rapid (and that number only grows if I also limit rapid to a lower rating). It is admittedly not conclusive as the two distributions are different, and I might explore the question further by comparing blitz players’ rapid improvements and vice versa but what I am seeing here certainly suggests that your improvement might be slower with blitz.</p>

<h4 id="limitations">Limitations</h4>
<p>The biggest limitation is that we are simply comparing the improvement of d4 and e4 players a year later but can’t know if the improvement is due to their choice of opening or because the groups of e4 and d4 players are different themselves - our study is simply correlattional.</p>

<p>It’s also important to note, that some in the ‘play longer games’ camp recommend going for games even longer than rapid but if longer games lead to more improvement I’d expect to see a difference in rapid vs blitz already.</p>

<h4 id="conclusion">Conclusion</h4>

<p>Longer games or at least rapid seem plausibly better for your improvement over blitz games (at least at lower ratings). d4 in particular seems to lead to a faster rating rise than e4 BUT the difference is fairly small, so if a player prefers a particular opening and would play more games using it then it’s unlikely to be worth switching. Personally, I will attempt to switch away from my default - the Scotch - if only because it would be somewhat silly to have ran this analysis and not changed anything because of it. If anyone wants to try to replicate my findings the quick code I wrote is <a href="https://github.com/Tenoke/e4_d4_analysis">here</a>, however, you’ll need to download the PGNs from Lichess as a month’s worth of games is ~200gb uncompressed and hard to share. Let me know if you get different results or spot any mistakes.</p>]]></content><author><name></name></author><category term="chess" /><category term="analysis" /><category term="e4" /><category term="d4" /><category term="blitz" /><category term="rapid" /><category term="shorter" /><category term="longer" /><category term="games" /><category term="lichess" /><category term="beginners" /><summary type="html"><![CDATA[A common question in the chess community is whether to play e4 or d4 as a beginner. Opinions are generally split between e4 being better for learning, no difference, and rarely that d4 is better. Typical reasoning being that e4 leads to sharper games and gives you more opportunities to learn. The other starting moves as white are rarely recommended for beginners altogether and rarely played at all levels. Another common question is whether playing shorter or longer games helps you improve faster, with the orthodoxy being that longer games help you learn more overall. The standard argument I’ve seen is that you have more time to analyze things in longer games which helps in all modes while the skills you improve in quicker games are less applicable in the longer time controls. I’ve downloaded games from Lichess’ database to explore those questions.]]></summary></entry><entry><title type="html">As Many Times As It Takes</title><link href="http://localhost:4000/fiction/as-many-times/" rel="alternate" type="text/html" title="As Many Times As It Takes" /><published>2021-07-21T00:00:00+02:00</published><updated>2021-07-21T00:00:00+02:00</updated><id>http://localhost:4000/fiction/Many-Times</id><content type="html" xml:base="http://localhost:4000/fiction/as-many-times/"><![CDATA[<p>I’m a wanderer now. My name is John Minter, my original world no longer exists, and I don’t grieve for it. What’s real and more importantly what’s going to be real is what matters. Did I kill Hitler? Of course. What self-respecting time-traveller wouldn’t? The very first time I told someone about my predicament they immediately asked why I hadn’t and it admittedly made me uncomfortable. So I did kill him. Twice in fact. ‘For the memes’ is what I wish I could say but that saying no longer exists and nonetheless, it’s not true. I didn’t even do it personally - I made changes to the past which indirectly caused him to die. Apparently, he lived the kind of life where dying prematurely was a real possibility. It does improve the world, I think, but as per usual it’s hard to tell.</p>

<p>How often have I travelled? 89 times. Not so many in the grand scheme of things, not so few that it hasn’t changed me either. The machine should be good for at least a few thousand more and the mission is far from completed. That number is bound to increase significantly.</p>

<p>When analysing a coordinate the machine only sends a handful of particles and records their response, yet even that act changes the world irrevocably. At least if the analysis is of the past.  So I don’t use it backwards as I can’t even know what has changed but I have it on good authority that the person that starts an analysis into the far past is extraordinarily unlikely to be the one to receive the results. Frankly, I’d consider it bad UI to even give me the option but they didn’t want to limit me. I don’t think it matters that I would most likely erase myself (and everyone born after the machine analyses) out of existence if I do it that way though. I wouldn’t have passed the psych evaluations if I was the kind of person who did. What does scare me is that due to random chance the resulting timeline after a travel or analysis will be ruinous and without anyone to change it. It’s a small risk but an utterly terrifying one.</p>

<p>Do I miss my original timeline? Not really - we were on the brink of apocalypse and everyone on the team knew they’d be erased as soon as I travelled back. Having some form of humanity is more important after all. We thought the fallout we got was inevitable. It seems we were wrong - I’ve only encountered a similar biochemical event one other time in my travels and even then it wasn’t quite as bad as what we faced. I prevented our whole issue on my first try! It was damn trivial! You’d think my job would be done once that was over but no, turns out that humanity doesn’t fare so well even without that hard Apocalypse. Societal and technological collapses are frequent. As far as I can tell this last timeline had those responsible for the worse of it lauded as heroes. Again. “They made things better”. They didn’t. Sometimes I wish I could be tempted to travel back a million years and do a hard reset on humanity. It would be irresponsible. I wouldn’t fit with the new humanity. I won’t even speak the languages. And they’ll likely only get one shot at it. I don’t venture before the 20th century - the further back I go the more changes there are and the less useful my knowledge is.</p>

<p>Even now I regret going back to 1910. Things are too different already - all the important people and events that happened later just don’t exist here. I believe I slowed down computer science by erasing Turing. Of course, It might’ve been due to someone or something else. That’s partially fixed but if I go to the early 20th century I’ll erase the current alternative technological path, too.</p>

<p>The other direction is no better. I don’t go too far forward because going to a point when a time machine is invented can be disastrous. Not because of any interactions, it’s all just normal matter. I can do whatever depraved acts I want with myself or my predecessors. Wouldn’t mean a thing. The real potential problem with the far future is that there might be others changing the timeline and that can impact my own interventions. I think.</p>

<p>At least there are less issues with letting the machine give me its basic analysis of the point of arrival in a given future. That’s what I am currently doing - firing off scans of the random points in this timeline’s near future and so far I like what they suggest. I can’t tell all that much more than the type of matter at the given locations, but I can find signs of humans at a reasonable frequency in major cities, normal levels of radiation and generally everything pointing towards there being no crisis any time soon. I’m currently in the year 2000 - a year I often use to gauge things as it’s around the time when my world was still doing okay and a nice round number. The first thing I noticed is how humid it is. I expect to remember that as a key feature of this timeline despite it likely simply being a feature of the particular day I traveled to. It’s just how it is.</p>

<p>They are pretty advanced here already. Whether it is due to my last intervention in 1959 or due to random chance I can only guess until I read up. A typical personal computer has 8-12 cores and over 50(!) gigabytes of their equivalent of RAM. A typical working week is 4 days, and only 1% (or 3.5% according to the secondary political party’s metrics) are in poverty.The average life expectancy is at 80.2 years. For some conflicting reasons wearing yellow is strongly discouraged (though not actually illegal in most jurisdictions) but hey, you get used to random things like that. I assume, say,  the prevalence of denim - a textile which I haven’t seen in a while - at this point in my timeline would be equally odd to them.</p>

<p>This world does not meet all my requirements - the worldwide incarceration rate is at over 1% and capital punishment is very well received but I’ve learned not to judge too much even when I don’t quite agree. I’ve accepted that the better world wouldn’t look quite like I’ve imagined. At any rate, the more data I gather the more prepared I get to laxen my Utopia requirements. Not yet, of course - there is more to try first. I do need to weigh the chances that I will perish in a bad timeline against the chances of successfully finding a good future.</p>

<p>I will travel to this world’s future to see it, maybe go back and forth a few times to reset or look for points where I can improve it further. There’s hope. I won’t go immediately of course. The machine takes a toll on my body - not as much as something like a space flight but enough, so it is wise to take breaks in between trips whenever possible and to keep in shape. I’m prepared to spend as many years as required on my mission after all - better stay sharp. Pawn shop first, then finding a place to stay and reading up on the effects of my last intervention. The usual.</p>

<p><em>Samuel Brosnan was currently the Chief Technology Advisor of the USA.</em></p>

<center>***</center>

<p>This future is unacceptable. Dystopia of a different colour. People get chipped as soon as they commit a crime and almost everyone has one now. You can’t tow the party line and despite the benefits of a unified world government this is no Utopia-adjacent world. I’ll need to travel before I’m found out - a back to back trip. Unpleasant. The mid 90s perhaps before going back to the 21st century and checking it again.</p>

<p><em>Samuel Brosnan died 11 years ago.</em></p>

<center>***</center>

<p>I’ve been traveling back and forth within this timeline, never earlier than 1989, never later than 2028. There’s much to learn here and occasionally I feel like I’m on the cusp of grasping it. Usually the timeline is ahead technologically, other times socially and sometimes both but there are always problems. I’ll need another try - probably a few - to start really figuring it out.</p>

<p><em>Sam Brosnan was President. Magnate. Man of the Year. The cliche hero engineer who devoted himself to science. On benefits after being assaulted. 3rd richest man. Leader of the IN reforms. Dead.</em></p>

<center>***</center>

<p>Bad luck. It was bound to happen. I traveled to 2021, but the government was after me so I immediately traveled back to 2003 in fear of getting found out again. The stress on my body was already too much and the rowdy drunks who attacked me caused some damage. My knee might not recover fully and this area is already advanced enough that seeking real medical help under the radar is not so easy. I yet again wish I hadn’t had to erase that one identity I prepared in the 80s all those timelines ago. I might need to wait to recover enough for a trip further back where seeking help is trivial. I hope I don’t have to. In the meantime it’s time to examine my options once again.</p>

<p>One option is to travel back and release blueprints and papers to increase the rate of technology advancement. This has not worked as well as I hoped before but I can try different variations. There are many possible variations of this. Many possible variations of everything. Too many, as it always seems to be the case.</p>

<p>Another one would be to micromanage by boosting certain people and advances and keeping others down. It can work but it is finicky. There’s more risk for myself the more involved I get. Still, I can try to identify more promising points for intervention. Sometimes it doesn’t work like my early misguided attempts with Turing - trying to help him avoid punishment for his sexuality helped no one. I do miss him. Sometimes it seems to help like with that recent Bell Labs equivalent.</p>

<p>Yet another alternative I’ve perhaps dismissed too hastily and which starts to look more tempting is to find smart people, get them on my side, tell them everything and work together.</p>

<p><em>Sam Brosnan was the prolific ‘investor poet’.</em></p>

<center>***</center>

<p>I had to spend over a month to heal enough for the series of jumps I needed but by the time I arrived in 1999 I had most of the information on how to quickly set up and get some of the best treatment available. I know a lot more now. Further, I know a lot more about now, who to contact and how. What’s left is to wait things out the old-fashioned way.</p>

<center>***</center>

<p>“topic: 3446124419567 ∑##$ zzrtrt ; no harm</p>

<p>Salute Brosnan,</p>

<p>You must be currently investing in $WWZ or writing your essay on bias in technology forecasting. I wouldn’t want to make you abandon either but we need to talk. I believe you secretly think you are on your way to becoming one of the most important people alive today. You are correct but let’s not set our standards too low. You are about to become much more than that.</p>

<p>344626 tonight. Message me when you confirm.</p>

<p>I’d tell you that you aren’t system infiltrated and my information comes from elsewhere but I understand you aren’t the kind of person to ignore the chance you are.”</p>

<p>This should do it. If it doesn’t - there’s always next time.</p>

<center>***</center>

<p>It didn’t. It took 17(!) more messages to get there but we are finally meeting. I’ve already paid off people in law enforcement - specifically those who were later caught for being less than exemplary, set up a driver nearby, emergency fund and a few other countermeasures in case Samuel ends up screwing me over. It’s time. Then again, it always is.</p>

<center>***</center>

<p>I am now facing the unexpected. My name is Samuel Brosnan, the world needs to be improved, and the tall, well-toned and impeccably dressed man in front of me claims to have the means. The same man who knew portions of my secret passwords - kindly rearranged either in case someone saw the messages, or because that’s all he could get, and other details I would have thought near impossible to know or guess.</p>

<p>I have checked all my records, and those of the people around me and there are no obvious security breaches, no missing hours, nothing. Computer system infiltration is always possible of course, but in his very first letter he assumed I was working on an essay which I had only started composing in my head and planning on writing all at once that night. I did not - my time went into planning how to approach this. I followed with more messages asking for other types of verification, ‘idle’ discussion which he must have realized was just hunting for hints. I got the most by hinting that I am abandoning the essay. He knew too much. Of course, my transition into writing about problems in technology forecasting is perhaps somehow predictable even if the timing is so perfect. His use of the exact terms I planned to borrow from other sciences when hinting at how important he thinks it is what put me on edge. Then again, so did a lot of the other things he said and promised but that’s what really got me. All of it made perfect sense, especially the parts about the best directions in research and development. This is no crank or at minimum they are extremely knowledgeable. Knowledgeable enough to have little problem arranging a meeting with me through more normal means. Knowledgeable enough that I should have heard of him.</p>

<p>I’ve mostly ruled out the more plausible theories. The best ‘mundane’ theory is a combination of system infiltration + lucky guesses + extremely smart people working in tandem. The chance of getting so much right is far too unlikely but perhaps if they did this to many others, lucked out only with me and had other sources of information.. I queried some influential people but none of them admitted to having received messages like this - “I wish the cranks would for once send me anything remotely plausible or interesting” is what Revan, the man I get most often compared to, replied.</p>

<p>The other theories which I have to consider at this point are much less likely a priori and are mostly different types of advanced technology. As unlikely as it is, if someone could do a full brain scan of me and select other people, parse that information and use it then they could do something like this. Same idea with a novel drug to make me tell them everything they need, then forget and somehow cover their tracks but there were too many logs they’d need to fake. If they had things like that, I do not know why they would even need me. Although, to be fair, I have little idea why they need me anyway. But I do want to find out.</p>

<p>I decide to open with a joke by leaning on one of my less likely theories. Lighten things up from the start, hopefully take control of the conversation.</p>

<p>“Salutations, John. It is clear you are a time traveler,” I say, smirking. It is important to look like you are in control even when you are not.</p>

<p>“I am, and you seem to be as smart or even smarter than you seem If you’ve already figured it out and accepted it. I assume then you already have a good idea of what we might be doing here,” he shoots back.</p>

<p>I most certainly did not and despite my competing urges to either drop my jaw or laugh, I stop to think. My first thought was on his accent. It was slightly off which was a small point in favour of him being a time traveler. As they say - the parallel is a different country. If he had brought it up, I would think even less of it as he is certainly capable of preparing that much but he did not. On the other side, it was now clear he had subtly tried to lead me towards that conclusion. ‘Today’ was emphasized in his first email. He indirectly talked about the future. He acted like he was both unknown and powerful. Regardless, I fall back on old habits and say the obvious - “Proving it.”</p>

<center>***</center>

<p>I listened to his explanation, asked my questions and told the presumed time-traveler to meet again in a day. It all mostly made sense but there were parts I could attempt to verify. His involvement in the 1949 shift for pre-registration of research and the related changes of typically used statistical methods. The dark matter anomalies or what would be anomalies under our current models. The theoretical validity of the various technology paths he claims we’ll be taking. Jana and Mellisa. I assume he realized I recorded it all.</p>

<p>It checks out as far as I can tell. It’s extremely unlikely but there are not many better explanations and I am not one to reject the unlikely if it fits. It’s not like I will stop looking for alternative explanations. I approach him.</p>

<p>He is smiling. His outfit is reminiscent of a security guard’s today. “Here’s a copy of your watch, that pretty logo from your lobby and Whales’ ‘Man versus God’. Lucky that you are fond of such a tiny painting. I can’t bring all that much with me,” he says.</p>

<p>I take my time verifying them, while noting the implication that he can only bring a little with him at a time. That’s fine and to be expected.</p>

<p>“Would my lab find anything unusual about those duplicates?” I ask idly to confirm my working model of his time travel. They should be completely normal matter. Really, I should be amazed about the implications of unlimited duplicates with minimal limitations but I am still preoccupied with being amazed with Time Travel in general.</p>

<p>“Not unless you analyze them with an IBM 5100,” he replies but quickly adds  “Sorry, a joke from another time. The reference means nothing now.”</p>

<p>It’s dangerous to make assumptions on a throwaway comment but I file this away. Something to investigate further. “And you can duplicate anything as long as it is small?”</p>

<p>“In theory, yes,” smiled the time-traveler. “In practice it is a bit more complicated with the biggest limitation being that the machine can’t travel to points in space-time that are too close together, at least not until the prior travel has been overwritten by one that is sufficiently earlier.”</p>

<p>I wonder if he is bluffing. This should put me at ease as it implies that he cannot easily retry his conversations with me over and over but he could also be lying.</p>

<center>***</center>

<p>I kept thinking how ripe for abuse the duplication abilities of the machine were - take something valuable, send it to the past, rinse and repeat and you have many copies of it. Apparently, you can’t travel to points within a year or so of each other but you can still work around it if you have a longer time frame.Cloning powerful chips or parts can come in handy but in the end it seems more likely to be a ‘step two’ in reaching what John calls ‘Utopia’. First, we need society to be mostly in order for enough years and then we can think about duplicating whatever we want to move to a post-scarcity scenario.</p>

<p>“I think you are approaching this all wrong. There is a better way,” I tell John directly while projecting that I mean it.</p>

<p>It is absolutely incredible that Time Machines were invented and everything didn’t dissolve into utter chaos immediately. Having the only person who has access to one be someone attempting to prevent the worst outcomes is a godsend. It’s still dangerous of course - any one man can and will err. Everything riding on their judgement and luck. I can only imagine how bad it must have been for those smart enough to build it to take such a risk. It is horrifying in a lot of ways but nowhere near as bad as the alternatives. If anything it’s surprising how fortunate we are.</p>

<p>“You already have a solution? I am skeptical,” John says but I could see the hints of hope and excitement on his square face.</p>

<p>“Yes or at least a much better approach. You have a blindspot.” The skepticism shows now and too much hinges on the next part. He’s not going to like it but I continue nonetheless “Of course, I have a few conditions first.” I tell him</p>

<p>“Go on,” he replies.</p>

<p>“I don’t get completely erased,” I say. “We leave the timeline before 1976 stable.”</p>

<p>“It is hard for me to relate anymore. I have accepted that my life is forfeit aside from this but I can understand how it must be different for you. I will warn you though - even the tiniest chance of a success is worth uncountably more than our lives are. I will have to think about it,” he says with a look betraying sorrow.</p>

<p>“Why? This is not just self-preservation on my part.” - to a large extent it was - “You already told me the current branches hold promise. You’ve apparently seen multiple versions of myself committed to improving things. You told me I can be the ‘chosen one’, which admittedly almost made me walk away. This is not a boast but I am clearly an asset to the world. You think it might be selfish? I am willing to accept that my current self will stop existing, a sacrifice you yourself have never taken,” I reply. The last part might seem unfair but it frames the conversation in a manner I can use.</p>

<p>He regarded me with a dark look before saying “To put it bluntly, I am not banking everything on this despite how much time I’ve invested. If it doesn’t work I need to keep my options open. Promises are not something I break either - it was an explicit part of both my selection and training in the hopes that if there are other time travelers they’d be trained similarly and the chance of us working together would be higher. So I won’t make a promise I might not keep and I don’t want to limit my options without a good reason. I still want to hear you out so I implore you to put yourself in my shoes and think how little of a sacrifice one’s existence is.”</p>

<p>To put myself in his shoes? I idly wonder if that saying is from his timeline. This is a very favourable response despite him not agreeing yet, and if he means what he says about promises - which makes sense on the face of it.</p>

<p>I decide to reveal more “I understand but you’ve gotten a few things wrong, my ally. First, my solution will leave no room for other trips. It will be final. Second, you profess selflessness but you are the blindspot. You think you need to be the one making trips which is not the case,” I tell him.</p>

<p>He stands up and looks like he might be considering abandoning this. I try not to think of the ever present threat of having this conversation and his interaction with me erased altogether. Despite a twinge of guilt, which I’ve long ago learned to suppress, I continue “No, you are still not getting it. Nobody has to. You are wasting enormous amounts of your machine’s limited capacity to lug yourself around. Even if you weren’t, you can do what? A few thousand trips in your lifetime? At best? We can do a lot better.”</p>

<center>***</center>

<p>Two weeks had passed since I started convincing him and it was working. He was partly worn down and I leaned on it to exaggerate his mental tiredness of his situation. I emphasized how he wont have to travel and he will finally be a part of a world again. The world. Minimized the risks perhaps, maximized his role so it doesn’t seem quite so out of his hands. I used every other trick I had. Mainly though, I just relied on the plan.</p>

<p>Changes had to be made. The main part didn’t. The machine can scan a small amount of particles and bring back results. This costs it energy but less than a full trip and it doesn’t interfere much with repeat travel. So why not set up our own hyperspecific places to check on top of the general ones and automate it? If you can make it possible to judge the quality of the timeline based on the checks the rest is easy. Change something small or leave a small message, fire up many checks each of which will result in a different timeline branching off that change. Do it as an optimal stopping problem - first fire enough checks to establish a baseline, and then keep at it for as long as the machine has energy - which if optimized should let it evaluate at minimum trillions of random timelines - and stop on the timeline that the policy judges most likely to be optimal.</p>

<p>The message for the machine’s automated checks can’t be as complex as we’d like, and wouldn’t be so easy to set up. Very specific spots where we will leave a milligram of lithium in onel case, gold with specific concentrations in the best one and so on.</p>

<p>Of course, versions of me and him would need to evaluate the timeline we live in, based on all our pre-selected rules, which were developed by John as much as me. We’ll not only need to do that and leave the right particles in the right places to evaluate it but also set up everything else. Maybe we die in some of them and the only positive checks there are those outside our control - things like global radiation levels but that’s okay. It most likely will at least ensure that in the final timeline at minimum one of us is setting up signals indicating the timeline is good and the neutral indicators don’t have obvious flags. In the best case, which I am optimistic about based on John’s experiences, we end up with an ‘Utopia’, or at least a timeline that we can make it one.</p>

<center>***</center>

<p>It has only been six months and the setup is complete. Everything is in place. The machine is modified. I did most of the work after a point but never without John’s supervision. It was only natural.</p>

<p>He is staring at our setup, rehearsing. “I will travel to June 1st 1989, a promising starting point. I will get out of the machine, ensure that I’ve set myself up properly and on July 20th I will initiate the process. This should be the last time I see the machine. I will not know whether this is the final timeline or a fleeting one, and will start contacting your 13 year old self using the scripts you wrote. By the time you are 16 I will have revealed everything and we will set up the first check in the High New England lab, which is largely funded by your aunt, Fernanda Brosnan,” he says.</p>

<p>“The plan is hundreds of pages long, it might be better to rehearse the contingencies than the base parts, my ally,” I tell him frankly. I should be more excited but all the work is done and I will in many ways stop existing. That simple fact is affecting me much more than I expected, not that I will say so out loud. I feel hollow. Better if that part of me is not something he remembers and carries to Utopia.</p>

<p>He asks me some more inconsequential questions and I do my best to answer them again. His future selves will still be a continuation of his current self so it’s exceedingly important to keep him grounded. My alternate selves will branch from how I was at 13. It should worry me that they won’t be as capable as me but I know from John that I consistently succeed across timelines and can only do better with guidance. No, what I am afraid of isn’t if they’ll do well. I worry if they’ll really be me, the me of right now rather than someone too different to really count.</p>

<p>“This is it, huh? My last go. We’ve been preparing for a while yet it still feels so abrupt. It’s so.. <em>final</em>.” John  says. For a moment I expect him to say more, but he just walks to the machine, my existence already irrelevant to him.</p>

<center>***</center>

<p>I am selfish. My name is Samuel Brosnan, and my world is not Utopia. I subverted John when I was 17 and faked all future checks. I did not want to be erased so I ensured my timeline will be the final one.</p>

<p>It sounds like the version of me that planned this didn’t want to be honest with himself and admit his fears. It was inevitable for one of us to take over - self-preservation is too strong a motivator. Did he truly not realize? Or did he realize and judged it best if I am the one in charge nonetheless?</p>

<p>Regardless, I’ll make it right. Surely using all the knowledge I have now and the dearly departed John’s resources will be enough. Enough to recreate the Machine. Enough to reach Utopia. I don my yellow jacket in defiance and get back to work. I can fix this.</p>]]></content><author><name></name></author><category term="fiction" /><category term="time-travel" /><category term="short" /><category term="story" /><category term="sci-fi" /><summary type="html"><![CDATA[I’m a wanderer now. My name is John Minter, my original world no longer exists, and I don’t grieve for it. What’s real and more importantly what’s going to be real is what matters. Did I kill Hitler? Of course. What self-respecting time-traveller wouldn’t? The very first time I told someone about my predicament they immediately asked why I hadn’t and it admittedly made me uncomfortable. So I did kill him. Twice in fact. ‘For the memes’ is what I wish I could say but that saying no longer exists and nonetheless, it’s not true. I didn’t even do it personally - I made changes to the past which indirectly caused him to die. Apparently, he lived the kind of life where dying prematurely was a real possibility. It does improve the world, I think, but as per usual it’s hard to tell.]]></summary></entry><entry><title type="html">The Unwatchability of Yesterday’s Videos Today and the Promise of Tomorrow</title><link href="http://localhost:4000/blog/lowres-highres/" rel="alternate" type="text/html" title="The Unwatchability of Yesterday’s Videos Today and the Promise of Tomorrow" /><published>2020-07-08T00:00:00+02:00</published><updated>2020-07-08T00:00:00+02:00</updated><id>http://localhost:4000/blog/AI-Superresolution</id><content type="html" xml:base="http://localhost:4000/blog/lowres-highres/"><![CDATA[<p>A topic I end up discussing with friends and thinking about regularly is how videos and games we used to like a decade or two ago looked completely fine back then and are currently unwatchable. I don’t mean this in the more common way of shifting tastes and values but in almost literally barely seeing what’s going on. Videos in 360p and even lower (how is 144p even still an option in youtube?) seemed complete fine to me but now they are not and I am not even able to make out the details. Games that looked like huge graphical improvements at the time and giving us a taste of how realism might look like soon now look absolutely awful to the point where I can’t even tell what’s going on with certain objects (if I can even recognize them) in some scenes. One might say “Well, you are spoiled by much better graphics” and sure, it does seem like it but what does that entail? Why did I use to think I can see things I now can’t?</p>

<p>My TV was definitely worse quality than now (if we pretend I even have one) but I am not sure if it was quite 240p. Reading <a href="https://en.wikipedia.org/wiki/High-definition_television">various</a> <a href="https://www.quora.com/What-are-the-resolutions-of-TV-sets-throughout-history-HD-then-1080p-then-4K">links</a> suggest 720p and more were already available in the 90s/early 2000s, at least <a href="https://en.wikipedia.org/wiki/Standard-definition_television">~480p+</a> seems to have been somewhat more common and at minimum movies at the cinema had a decent effective resolution by then even accounting for the distance. Yet Youtube <a href="https://en.wikipedia.org/wiki/YouTube#Quality_and_formats">only offered 320x240</a> between 2005-2008 which was mostly fine then and nearly unwatchable now?</p>

<p>There is a popular <a href="https://www.youtube.com/watch?v=dMH0bHeiRNg">Evolution of Dance</a> video from 2006 which I remember and in my memories, it looked pretty okay. I just re-watched it now and sure, I can still tell what’s going on but I can, for example, barely see the dancer’s face. Indeed, it isn’t that bad if you watch a single clip, however, when I tried to watch an <a href="https://www.youtube.com/watch?v=ME7smdgMVr4&amp;list=PL1E7ADC0B6C30C1DF">older sketch show</a> for more than a couple of episodes the quality made it actively unpleasant. The effect also definitely exists in games that seemed to look pretty good to me - e.g. <a href="https://www.google.com/search?q=command+and+conquer+old&amp;safe=off&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;biw=1252&amp;bih=912">Command and Conquer</a> which we will get back to later or <a href="https://www.google.com/search?q=Tomb+Raider+3&amp;tbm=isch">early Tomb Rider games</a> for a 3D example.</p>

<p>One argument is that old videos look worse because of our massive screens today. Checking on my phone there might be some truth to it, however, I am far from convinced that it explains it.</p>

<h4 id="refresh-rate-comparison">Refresh Rate Comparison</h4>

<p>People getting used to better quality and finding it hard to go back is not uncommon of course - I hear it quite often today in regards to higher refresh rate displays. You get comments <a href="https://www.reddit.com/r/buildapc/comments/bgxf5w/for_those_wondering_if_a_144hz_monitor_is_really/">like</a> <a href="https://www.reddit.com/r/OverwatchUniversity/comments/5srzgk/does_a_higher_refresh_rate_on_a_monitor_really/ddhdotb/">this</a> claiming that <em>“You don’t realize how ‘laggy’ 60hz is until you go to 144. Now I can never go back.”</em> or <em>“now I HATE my monitors at work”</em> which I find interesting as the ‘never go back’ sentiment is very common. In fact, if you try to look into 144 Hz monitors these comments are nearly ever-present. This is seemingly more prominent with people who mainly use higher refresh rates, less so among those who use e.g. a lower refresh rate on one screen and a higher on another. That suggests to me that perhaps in general nearly constant exposure to better quality ‘ruins’ low quality much more easily than if you get exposed to both.</p>

<h4 id="hallucination-theory">“Hallucination” Theory</h4>

<p>An argument which I like is that we didn’t know much better but were used to those low resolutions and found it <em>easy to fill in the details</em>. The brain already does <a href="https://en.wikipedia.org/wiki/Filling-in">a lot</a> of <a href="https://en.wikipedia.org/wiki/Saccadic_masking">that kind</a> of <a href="https://en.wikipedia.org/wiki/Visual_release_hallucinations">thing</a> all the time, so it doesn’t seem so unreasonable. Additionally, most of us currently rarely watch anything in low-res and we are more inherently aware that a given thing is in bad quality when it is, due to our regular experiences with HD, so we wouldn’t normally be benefitting from that particular extrapolation. I wouldn’t be surprised if this is partially what’s happening in addition to a more general “we just didn’t know better” explanation. People with <a href="https://en.wikipedia.org/wiki/Anton%E2%80%93Babinski_syndrome">Anton’s blindness</a> can fill in <em>everything</em> and think they are seeing while being completely blind, what is some minor upscaling and ‘hallucinating’ more details after you get used to looking at only low-quality videos and games on your computer or TV since you were born.</p>

<p>Regardless of whether the above explanation is correct, where does this leave us? It seems clear that we could enjoy these lower in visual quality works before but a lot of us can’t do so quite as easily today. Should we consider re-training ourselves at enjoying those works? That seems a bit tedious. Is this just a (small) price we have to pay for progress and call it a day? Maybe, but it seems like progress might still have us covered. There have been a lot of advancements in AI Super-Resolution and we are getting to the point where it can be used well in <em>some</em> cases, and things like games and old videos that aren’t in <em>too</em> bad a quality can benefit a lot from it.</p>

<h4 id="ai-super-resolution">AI Super Resolution</h4>

<p><em>Of course, it will be irresponsible not to mention that upscaling and enhancements don’t have their issues. It is impossible to accurately increase the quality of an image without guessing at what should be in the extra pixels. This is typically done by showing a lot of examples to the model and it learning the most common ways to upscale accurately - but if you are upscaling something too different from what’s in the original dataset, <a href="https://www.vice.com/en_us/article/7kpxyy/this-image-of-a-white-barack-obama-is-ais-racial-bias-problem-in-a-nutshell">there might be issues</a>. This is part of the reason why fully-automated upscaling without humans in the loop for serious tasks beyond entertainment can be unwise.</em></p>

<p>Games, in particular, have benefited a lot from AI-based upscaling - for example, Nvidia’s <a href="https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/">DLSS</a> which is used more and more or Facebook announcing their own (near) <a href="https://research.fb.com/blog/2020/07/introducing-neural-supersampling-for-real-time-rendering/">real-time Neural Supersampling</a> using additional data available in Game Engines. Even more relevant, the 1995 Command and Conquer game, which looked pretty good to me as a kid then and awful to me as an adult now has been <a href="https://www.ea.com/games/command-and-conquer/command-and-conquer-remastered?setLocale=en-us?setLocale=en-us">recently remastered</a>, largely using AI to great success! This includes both the playable portion of the game but also the videos and sounds in it. As far as I can tell, this has been met with overwhelming approval and the game is currently in <a href="https://store.steampowered.com/search/?filter=topsellers">Steam’s top-seller list</a>. Of course, this is far from the first remastered game - in fact, the practice is getting more popular - but it uses AI for it’s upscaling to an even larger extent than those before. That and the general trend of reworking old games can paint an optimistic picture where in the near future we might be able to get most old games brought back to life with significant visual improvements and less and less work for the developers.</p>

<p>What about videos in general? <a href="https://www.youtube.com/embed/3RYNThid23g">This video</a>, originally from 1896 recently made the rounds when it was upscaled to 4k, 60 fps and I have to say - it looks a lot better to me.</p>

<p><img src="https://www.sciencealert.com/images/2020-02/4k-train_1024.gif" /></p>

<p>There are <a href="https://www.youtube.com/results?search_query=upscaled+video+ai">many other examples</a>,  although currently more often than not you still need a human guiding the process to make sure you end up with a version that doesn’t have too many <a href="https://www.youtube.com/watch?v=XZVxiak0axo">unpleasant artefacts</a>. Additionally, the process is still very compute-heavy and far from real-time. Either way, as algorithms get better, datasets larger and compute cheaper it is likely that in a few years we’ll be able to upscale any old video on demand or at minimum that most older content that is at least somewhat popular will eventually get the super-resolution treatment. Things like browser extensions and apps that do it for you automatically and in real-time might soon become common for those who want them.</p>

<p>For the time being, you can pay someone to upscale a video for you, e.g. <a href="https://neural.love/">neural.love</a> who made that 1896 train video, you can register for <a href="https://deoldify.ai/">various services that deoldify photos</a> or you can <a href="https://colab.research.google.com/github/tg-bomze/Face-Depixelizer/blob/master/Face_Depixelizer_Eng.ipynb">play with</a> <a href="https://github.com/idealo/image-super-resolution">models</a> directly in Colab or on your own machines. The future looks bright and with the right model so will the past.</p>]]></content><author><name></name></author><category term="machine-learning" /><category term="AI" /><category term="image" /><category term="video" /><category term="games" /><category term="super-resolution" /><category term="super" /><category term="resolution" /><category term="upsampling" /><category term="zoom" /><category term="enhance" /><category term="deoldify" /><summary type="html"><![CDATA[A topic I end up discussing with friends and thinking about regularly is how videos and games we used to like a decade or two ago looked completely fine back then and are currently unwatchable. I don’t mean this in the more common way of shifting tastes and values but in almost literally barely seeing what’s going on. Videos in 360p and even lower (how is 144p even still an option in youtube?) seemed complete fine to me but now they are not and I am not even able to make out the details. Games that looked like huge graphical improvements at the time and giving us a taste of how realism might look like soon now look absolutely awful to the point where I can’t even tell what’s going on with certain objects (if I can even recognize them) in some scenes. One might say “Well, you are spoiled by much better graphics” and sure, it does seem like it but what does that entail? Why did I use to think I can see things I now can’t?]]></summary></entry><entry><title type="html">StyleGAN for Evil: Trypophobia and Clockwork Oranging</title><link href="http://localhost:4000/blog/stylegan-for-evil/" rel="alternate" type="text/html" title="StyleGAN for Evil: Trypophobia and Clockwork Oranging" /><published>2020-03-27T00:00:00+01:00</published><updated>2020-03-27T00:00:00+01:00</updated><id>http://localhost:4000/blog/Stylegan-for-evil</id><content type="html" xml:base="http://localhost:4000/blog/stylegan-for-evil/"><![CDATA[<p>GANs have become the default image generation technique, and many are familiar with sites like <a href="https://www.artbreeder.com/">artbreeder</a>, <a href="https://www.thispersondoesnotexist.com/">thispersondoesnotexist</a>, and its off-shoots such as <a href="https://www.thiswaifudoesnotexist.net/">thiswaifudoesnotexist</a>. The generated samples are stunning and quite hard to distinguish from real images, and StyleGAN, in particular, has been used and finetuned by many with interesting results.</p>

<p>I’ve experimented with them before - for example, the projection of real images onto the StyleGAN latent space, and then modifying those is something I find quite interesting. In this project, however, I decided to go in a different direction and use them for evil.</p>

<p><strong>All generated horrors in this post are hidden unless you click on them. You can read without looking if you so choose</strong></p>

<p>Now, some of the more traditional ways of using Generative models for evil are for spam, <a href="https://www.wired.com/story/deepfake-porn-harms-adult-performers-too/">deepfakes</a>, <a href="https://newsyoucantuse.com/">fake news</a>, and <a href="https://analysis.leadstories.com/3471185-fake-faces-people-Who-Do-Not-Exist-Invade-Facebook-To-Influence-2020-Elections.html">scams</a> but we are going in a different direction here.</p>

<h3 id="trypophobia">Trypophobia</h3>

<p><a href="https://en.wikipedia.org/wiki/Trypophobia">Trypophobia</a> is a common condition where people react with discomfort and even strong disgust towards specific irregular patterns of holes or bumps. For some the effect is quite strong, for others, it is just unpleasant, and some like me can see what the issue is but have almost no negative reaction.</p>

<p>It so happens that there is a small, but just <a href="https://www.kaggle.com/cytadela8/trypophobia">big enough dataset on Kaggle</a> to train a StyleGAN on.</p>

<p>Here are a few examples of how the dataset looks:</p>

<details>
<summary>
<i>Example dataset 1 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/dataset1.png" /></div>
<br />
</details>
<details>
<summary>
<i>Example dataset 2 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/dataset2.png" /></div>
<br />
</details>
<details>
<summary>
<i>Example dataset 3 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/dataset3.png" /></div>
<br />
</details>
<p><br /></p>

<h3 id="training">Training</h3>

<p>The training data is very varied - ~6000 images many of completely different objects that only share somewhat similar hole patterns. Inconsistent data works poorly with StyleGAN - what works best is things that are cropped in the same way, from the same angles and generally the same thing but with different features - e.g. faces, even doing full bodies often lead to blobs (less so with stylegan2). This is certainly something that should be a problem with our dataset, but on the bright side we don’t care if the rest of the image is blobby, we just want it to learn the specific patterns that trigger trypophobia, doesn’t matter if the rest of the image is abstract - in fact, it is better in some ways if it is abstract, as that can make the videos and images more appealing.</p>

<p>I took one of the pretrained StyleGANs nearly at random - churches and started finetuning on the dataset.
The results even early on looked quite interesting. Here are some from the first hours:</p>

<details>
<summary>
<i>Example training 1 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/training1.png" /></div>
<br />
</details>
<details>
<summary>
<i>Example training 2 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/training2.png" /></div>
<br />
</details>

<details>
<summary>
<i>Example training 3 <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/training3.png" /></div>
<br />
</details>
<p><br /></p>

<p>If you find the above fascinating, you can see <a href="https://mega.nz/#!eVsC0AjC!6QHzPJMlwuHm5mkxTPOhGsII2ZGycQAv2hE9up6JZYc">much</a> <a href="https://mega.nz/#!KMtljazI!R4KrNNJMsx4wn4OF8L7gf-TQoakTVee3BO-Myfrq4S4">more</a> <a href="https://mega.nz/#!nV0zAKgZ!p-0qpKluSlDkTHfjRHR-IYtEnD5zs3RT5QJqvoZdGME">at</a> <a href="https://mega.nz/#!HU9wXYzL!DAjf448jAXej_38vmmSArDtAV3XHZLXqXvw092DtTBY">these</a> <a href="https://mega.nz/#!WVU12ayB!sNs5_BgT4iW9CTGtUt4QcI9r8ny83eBMvOvaOGHCaOg">links</a>.</p>

<p>Here are some video versions.</p>

<details>
<summary>
<i>Example vid 1 <small>(click to expand)</small></i>
</summary>
<br />
<iframe width="560" height="315" src="https://www.youtube.com/embed/l_gK9Q__GVY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<br />
</details>
<details>
<summary>
<i>Example vid 2 <small>(click to expand)</small></i>
</summary>
<br />
<iframe width="560" height="315" src="https://www.youtube.com/embed/yj_mAhj2EwY" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<br />
</details>
<details>
<summary>
<i>Example vid 3 <small>(click to expand)</small></i>
</summary>
<br />
<iframe width="560" height="315" src="https://www.youtube.com/embed/vvww_rZQVsI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<br />
</details>
<p><br /></p>

<p>Responses of the above vary - some are too disgusted too quickly, while others get just a bit of disgust from the worse of it while being intrigued overall, and some don’t get a negative reaction at all.</p>

<p>I should also note - videos like these might also be useful for good, as a form of <a href="https://en.wikipedia.org/wiki/Exposure_therapy">Exposure Therapy</a> but let’s not focus on that.</p>

<p>All of the above was trained on Colab using a p100. The final model was trained for roughly 4 days (I saw no improvement near the end), but there were good results within the first day of training.</p>

<h3 id="clockwork-oranging">Clockwork Oranging</h3>

<p>After looking at the early examples of my trypophobia finetunning, I saw some churches that still very much look like churches while also having trypo features. For example these:</p>

<details>
<summary>
<i>Church AND Trypo <small>(click to expand)</small></i>
</summary>
<br />
<br />
<div style="text-align: center;"><img class="img-fluid" src="/static/trypo/trypochurches-training.png" /></div>
<br />
</details>
<p><br /></p>

<p>This gave me the idea that you can fairly easily combine the two (or combine trypophobia and similar images with something else) to create videos which can make someone associate a concept - in this case a church - with negative feelings a la <a href="https://en.wikipedia.org/wiki/A_Clockwork_Orange_(film)">A Clockwork Orange</a>. I chose to add churches to the mix because that’s what my pretrained model was already trained on, and I had the dataset (it was not a religious statement) .</p>

<p>The idea (in this scenario we are very evil) is to strap someone (with trypophobia) to a chair and force them to watch those videos which only require some flowing patterns, at least some churchiness and at least some trypophobia-triggering. Admittedly, those videos didn’t turn out quite so well - I had to experiment with training them with different ratios of trypo to church photos, but they (and the possibly better photos above) demonstrate the idea well enough.</p>

<details>
<summary>
<i>Example vid 1 <small>(click to expand)</small></i>
</summary>
<br />
<iframe width="560" height="315" src="https://www.youtube.com/embed/KtZFkdP88tQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<br />
</details>
<details>
<summary>
<i>Example vid 2 <small>(click to expand)</small></i>
</summary>
<br />
<iframe width="560" height="315" src="https://www.youtube.com/embed/pvmtJTIKAls" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<br />
</details>
<p><br /></p>

<h3 id="conclusion">Conclusion</h3>

<p>This was a fun little project, that I spent a few days on. You would need to spend a bit more time to create something actually dangerous (which I doubt this really is) but not that much more. Of course, what I describe as clockwork oranging doesn’t look so useful in the real world but what if you could (and you can) use it to make some images just a bit more repulsive to the general population and others just a bit more appealing with little effort? That seems bad but then again a lot of advertising fits that mold despite it requiring more time per image/video. And at any rate - neither this nor ‘evil’ are near my main concerns with respect to machine learning. So even though I was thinking of finishing this with something about the dangers of AI, I’ll just let people make their own conclusions if there are any to make.</p>]]></content><author><name></name></author><category term="machine-learning" /><category term="training" /><category term="code" /><category term="StyleGAN" /><category term="image" /><category term="image-generation" /><category term="trypophobia" /><category term="clockwork" /><category term="orange" /><category term="conditioning" /><category term="evil" /><category term="gan" /><category term="disturbing" /><category term="phobia" /><summary type="html"><![CDATA[GANs have become the default image generation technique, and many are familiar with sites like artbreeder, thispersondoesnotexist, and its off-shoots such as thiswaifudoesnotexist. The generated samples are stunning and quite hard to distinguish from real images, and StyleGAN, in particular, has been used and finetuned by many with interesting results.]]></summary></entry><entry><title type="html">Training Google’s Reformer - takeaways, code, and weights</title><link href="http://localhost:4000/blog/reformer-99m/" rel="alternate" type="text/html" title="Training Google’s Reformer - takeaways, code, and weights" /><published>2020-03-18T00:00:00+01:00</published><updated>2020-03-18T00:00:00+01:00</updated><id>http://localhost:4000/blog/Reformer</id><content type="html" xml:base="http://localhost:4000/blog/reformer-99m/"><![CDATA[<p>A lot of progress has been made in training large NLP models in recent years, and after <a href="/blog/gpt-15b-chat-finetune/">finetuning GPT2-1.5b</a> I’ve been thinking about context windows. Large Transformers are very good, but when you are only limited to 512 or more commonly now 1024 tokens, the model doesn’t know quite that much of what happened earlier at any point. Google’s <a href="https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html">Reformer</a> paper introduces a more efficient way of doing attention, sidestepping the problem which I used to train a model with context of 64k tokens.</p>

<h3 id="reformer">Reformer</h3>

<p>The two main tricks used by the Reformer are reversible residual layers and Locality-Sensitive-Hashing.</p>

<p>Reversible layers lower memory consumption by allowing us to recalculate layer inputs during backprop based on the already computed later layers instead of keeping all of it in memory. It provides similar advantages to <a href="https://github.com/cybertronai/gradient-checkpointing">memory checkpointing</a>, except in this case the layers are designed to specifically capture the layer-to-layer changes allowing us to recompute more efficiently.</p>

<p>More importantly for my purposes, it also introduces Locality-Sensitive-Hashing (LSH), which allows us to approximately compute attention much more cheaply and thus work with much, much larger context. In short, LSH maps similar vectors together rather than working with all possible pairs of vectors (which is what makes attention so expensive). In the cases where we have many of the same/similar tokens, a lot of them will end up with the same hash so they get chunked and then attention is applied in parallel on those much smaller chunks.</p>

<p>If you are interested in the specific check out <a href="https://github.com/google/trax/blob/master/trax/models/reformer/reformer.py">their code</a>, <a href="https://arxiv.org/abs/2001.04451">paper</a> or <a href="https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html">blog post</a></p>

<h3 id="training">Training</h3>

<p>Google Research released an implementation of their Reformer model as part of the <a href="https://github.com/google/trax">trax library</a>, which is easy to use. Given my interest in NLP models with larger contexts, I decided to experiment with it and pretrain the largest model you can fit on a Colab TPU - with the idea that this way others can take it and finetune it on their tasks in cases where large context is important.</p>

<p><strong>Here is my related <a href="https://github.com/Tenoke/trax-reformer/">repo</a>, <a href="https://mega.nz/#!SR9WySCC!K2sdePWHDq2QiHlom8ZMLgKeBCcjQxsBToxl9NpNhpg">link to the weights</a> and <a href="https://colab.research.google.com/drive/1VSaQingkpMDf730ARlo2dw5vCBVOgt17">Colab</a> link</strong></p>

<p>First I used a small amount of the data to play with Reformer. I started with their <a href="https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb">code for training on Crime and Punishment</a>, however, I just couldn’t get it to go much beyond .3 accuracy no matter how much data I fed it. It seems like the parameters used there were mostly chosen for having a huge context but the model couldn’t learn much. After a lot of experimenting, I ended up with parameters closer to their <a href="https://github.com/google/trax/blob/master/trax/configs/reformer_enwik8.gin">enwiki8 training run</a>, I ran about 50 trials with small adjustments and chose the version of the parameters where the loss was going down quickest. You can see the final values I’ve chosen in the <a href="https://github.com/Tenoke/trax-reformer/blob/master/configs.py">repo</a>. <strong>The overview is that we have ~99m parameters (compared to the small gpt with 117m), context window of 65536, 0.2 dropout, 12 layers and a vocabulary of 256</strong></p>

<p>After that, I decided to train it on all of Wikipedia, which I downloaded and preprocessed to make it easier and faster to use with Colab and Drive. I downloaded the <a href="https://dumps.wikimedia.org/backup-index.html">most recent Wikipedia dump</a> and processed it into JSON while excluding all articles under 1000 characters - we are training for large contexts, so including tiny articles would be counter-productive. Then I split the big JSON file into smaller files of 5000 articles each - because Drive can lock down your file if it’s too big and you access it too much. <a href="https://colab.research.google.com/drive/15WCdIYju3A4UOY1-JGt7NTlw0jMLJYXZ">Here</a> is the Colab for doing the above.</p>

<p>Next, I wanted to pre-tokenize all my data so I can load it to my model quickly rather than doing it on the fly. One thing currently not added to the Reformer (but is on the list) is handling bigger vocab sizes, like the ~50k GPT2 vocabulary. Given that our context is going to be huge, and that the vocabulary size that enwiki8 config uses is 256 anyway, I decided on a character-level ByteLevelBPE tokenizer from <a href="https://github.com/huggingface/tokenizers">huggingface’s tokenizers</a>. I regretted it a bit, as there is little documentation in the library, and there are weird things in it like being unable to easily add extra tokens, or save the tokenizer - it took me a while to realize that saving after adding extra tokens does nothing, as they just save ‘the model’ as you’ve trained it, so I had to manually edit the vocabulary I pass to it. <a href="https://colab.research.google.com/drive/1letuN1YbhI8cxJIL059Bp7J6ga_kXkeP">Here</a> is the Colab for tokenizing and saving all the preprocessed data. Again - we are tokenizing in advance instead of on the fly to speed up training.</p>

<p>I wanted to take advantage of the large context so my function for feeding data to the model specifically always loads full articles and adds padding up to max the context length. The data is all tokenized and ready, so we just load preprocessed files at random and pick random samples from them.</p>

<p>After all that I trained it until the loss wasn’t going down for a while so I stopped a little after 125k steps. The accuracy at that point was a little over 0.75.</p>

<p>Here is how an example article from the dataset looks like:</p>

<details>
<summary>
<i>Example Training Article <small>(click to expand)</small></i>
</summary>
<br />
{"title": "Bill Bayliss", "section_titles": ["Introduction", "References"], "section_texts": ["'''William Bayliss''' (19 December 1886 \u2013 12 February 1963) was a British trade unionist.
<br />
<br />
Born in Leicestershire, Bayliss left school at the age of twelve to work at a colliery.  He joined the Leicestershire Miners' Association, but after becoming involved in industrial action, he was sacked and, a year later, moved to Nottinghamshire to find work.  There, he became active in the Nottinghamshire Miners' Association, and also in the Labour Party.
<br />
<br />
Bayliss became his union branch delegate in 1915, and served a year as vice-president of the union in 1927, and a year as president in 1929.  In 1932, he became the union's full-time financial secretary.  In this role he worked with Herbert Booth to promote reunification with the rival Nottinghamshire Miners' Industrial Union, which was achieved in 1937, whereupon he became an agent for the merged Nottinghamshire Miners' Federated Union (NMFU).
<br />
In 1943, Bayliss was chosen as one of the Trades Union Congress' two representatives to the American Federation of Labour.
<br />
Bayliss was elected to Nottinghamshire County Council, becoming an alderman, and serving as its chairman from 1945.  In 1946, he was elected as President of the Nottinghamshire Area of the National Union of Mineworkers, successor of the NMFU; he served until 1952, when he retired and joined the National Coal Board.
<br />
", "
<br />
"]}

</details>

<p><br /></p>

<h3 id="sampling">Sampling</h3>

<p>I got okay samples from it when they were less than a few thousand tokens long but if I tried to get a very long one it often either turned repetitive or turned into gibberish after a while. Part of it was because in the training data - ‘}’ denotes the end of a sample and it hasn’t seen anything after, so after it outputs that it just doesn’t know where to go, which is reasonable. However, part of it was just that it is easier to enter into repetitive sequences in longer samples as it has more options to do so.</p>

<p>For that reason I experimented with adding top_k, top_p, temperature, using exp2 instead of exp on the logits so they’ll be closer together, and a few random things including a Meena-inspired approach of generating multiple samples at each step and using the probabilities of the best one (also multiplying multiple samples’ probabilities). I’ve included some of the more useful ones in the sampling function in the <a href="https://github.com/Tenoke/trax-reformer/blob/master/sample.py">repo</a>. 
After adding sampling options, I generated 8 samples for various options (279 in total after excluding all options which were more obviously subpar) and wrote some testing code to show me one randomly and ask me to rate it. This was very tedious and I wish I had set more rigorous grading rules, as I did the grading over a few days and ended up not being as consistent as I would have liked. I’ve included the ratings of the settings I tested <a href="https://github.com/Tenoke/trax-reformer/blob/master/rating-args.dict">here</a>.</p>

<p>Here are various samples at different settings:</p>

<details>
<summary>
<i>Example 1 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "2000s", "2001s", "2002s", "2003s", "2004s", "2005s", "2006s", "2007s", "2008s", "2009s", "2010s", "2011s", "2012s", "2013s", "2014s", "2015 "2016", "2017 "1919", "See also", "References"], "section_texts": ["
<br />
<br />The '''Best Possible Choice of Discussion''' is a public biology focused on personal Strategy at St. Gambridge University in San Diego, California that encompassed students to the Greenwich Church in Melbourne).
<br />
<br />This is a third choice on a discussion in the former Greenwich church, however, it is noted for its original way to be Ethnographic and the Greenwich known as Greenwich MacLeod, its lineage. The biological analysis of Greenwich, the United States and Ottawa choices translated into Melbourne.
<br />", "
<br />As of July 1, 1997, the choice has a plantation service over the dozen St Gambridge Battle in Discussion, Melbourne, Perennial Bennett, Illinois, in addition to changes performances in Strategy, tax appearances, and cultivation. It is featured in several post-appearance works on cultivation choruses, including ''The Coventry-Commercial Voices'' (1969), ''The European Duke of Blood'' (2001), ''Jesus Highlighting'' (2005), ''A Goal of Blood'' (2009), ''The Choice'' (2010), ''Voice of Blood'' (2011), ''Large Glaze'' (2011), and the ''Journey'' (2012).
<br />
<br />The choice won their first First Appearance against Strategy at Blood's Third Appearance in 2012 and finally received the Cup Awards in 2007.  This public draws works from the Best Possible Choice to speak Live Arts and Best Speak Actor for the Year ''Voice'', and Co &amp; Bavarian Prize. In 2014, the choice was opened to 2,500\u00a0boys (75%)
<br />", "
<br />In August 2014, Best Possible Choice inked the comedy-first as a sales offense that would preserve its national appearances.
<br />Religious affection
<br />
<br />===Choice and Choice===
<br />The choice usually took place at the Football Hall of Fame in New York since 2000. The choice was sequestered by their position on June 30, 2003.
<br />", "
<br />
<br />Choice was founded in South Korea, and between 1996 and 2001 (with Alberto Chunter's areas being cast on Melbourne).
<br />
<br />===Mini-videos===
<br /> Journey together with others in Melbourne
<br />The choice was a recipient, and played on the Mini-video of the Year ''Voice'' (1996), with Deine Harrison of the Episcopal Church, and is the youngest part of the Choice and Choice position at the Choice Mini-video's parish.
<br />
<br />===Competitive show and performance in Finland===
<br />In March 2012, the first assembly versus the second assembly, ''Finternational Career''. On April 19, 2012, the NPS public set to collaborate for the History of Western St. Gambridge's, bonus transfer. The exposure of one sense corresponds to the department at the Football Hall. When Football Hall is called the ninth-legged F.A.N. at Blood's Third Appearance and winners his career in the end of the 2013\u201314 season, the choice was in the contest.
<br />
<br />==''Zeller North''===
<br />In January 2014, Ola Schmott. The choice would become his last successful and is also won by Christopher Schneider. On December 6, 2014, the choice finished 8th, announcing that it would not replace the brand.
<br />
<br />===Crew Diagnostics===
<br />The Crew Diagnostics included Crew Diagnostics, Bass Young and Drummond Diagnostics.
<br />
<br />The final session of Crew Diagnostics
<br />", "'''First Appearance Choice in Eastern St. Gambridge''', was released in 1996. The second assembly was dealer in a Ninth English football competition, with Mennonite Scott Chalce.  Sold in 2005, the Choice had a row with 11.5%. In the Melbourne Football League, the choice became a record of 120 or 18.4% of the vote. Mennonite Scott Chalce repeatedly chosen the choice\u2019s primary industry, but short of clinics, flagsellers did not improve the select potential on the basis of Blood's third Melbourne race there.
<br />
<br />===Infrastructure===
<br />In July 2015 \u00a7\u00a3240,000, Fontier Dr Boris, the choice was a friend of Robert Rogan and Dave Nicholson.
<br />
<br />On February 17, 2016, the Ola Liga of the Art of Fontier's Choice and the following of Competing Knights on December 37. 
<br />", "
<br />
<br />Spencer Fault hosted Rufous Hosts in the fall of 2018 and they made his own speeds as greatest choice for far-voice football players in Alberto.
<br />", "'''History of Western St. Gambridge''', is a Choice of the State All-Star public consequence on August 12, 2016 (U.S. Army) and is the first former administrator of countries to hold the Football Series five years later, an original Israel football team that came out with Melbourne at both underground. This time it was a popular veberation.
<br />", "
<br />===Competitive show and choice===
<br />In September 2013, the preceding choice announced nearly $100,000 to . 
<br />
<br />In the early 2016 Deiner Harris announced that it would be called the Football Choice. On May 27, 2017, the choice was announced. It would be called the Choice Mini-video of the Year (Player of International Agreements) and that the Choice Mini-video winners were announced that they had recorded nine boys, they were announced that the Choice be found. The Choice Mini-video was funded on June 30, 2015 by the Royal Magic. It is one of first segments to play a permanent role in the Army.
<br />
<br />In its Knights of Western St. Gambridge played for Live Arts. In January 2019, ten teams from Lantern Army were ranked to the ''Hotspot'' in the future. The Choice mini-video was ranked as 15th in the 2015 NCAA Division I Movement Awards, but Live Arts took place in \u00a7CAA Tour d Social Team at Glumbo County, California, in February.  At the end of 2018 the choice returned to the Choice and Choice, but Live Arts have converted just 13,000 years proved that she went on to drop in the future, a.k.a. City in Boston. Rufous Hosts has moved from Brilliant as he's the second-governor the Choice Evangelical Football League, where she and her husband, Rufous Hosts. When Rufous winning another an ASL that campaigned for the Year, she said \"It was going touching throughout soccer in Melbourne, where burghed into a museum in Manhattan. It was said that she \"repeated the buenos Publishers and whose book in the museum's halls about the architecture of the scotch and more future irregular ability to spread to them on a new, installation, multiple grape shock lines.\"
<br />Recording of the 2019 Western St. Gambridges, although speaking at the U.S. Army won the medal of the season.
<br />
<br />The artist received more than 750 unanimous players, and they became appointed One Football Conference (OFC), which included Conference Football League Football Conference (DFLFC), for future research international monitoring football teams. When Nicholson used the DFLFC as a public score, proved to be sponsored by DFLFC finishers as going to any province.
<br />
<br />===College career===
<br />The College career was opened on May 6, 2016, three weeks later in Shootout contesting Conference State All-Star End for managing Delaware National Team Graphic and the Choice Mini-video.
<br />
<br />Alchemy was launched in 2009, where Best Actor at Paris Football League, a slowly engaged actress, was stated in activities until choosing to acquire U.S. Army parties.
<br />
<br />Although they were translated into the U.S. Army, the choice was entirely repeated in APR before including the \"Coventry Choice\".
<br />", "===Crew Diagnostics===
<br />In January 2015, it was announced that the following of the choice's hall, Alberto Chunter, died on April 1, 2016. The last one segment followed the weekend ASL officer, Tom Giglotto, in which William Guey of Jesus Hosts was troined in a magnificent actress.
<br />
<br />The choice was adnated for Rufous Hosts which purchased the Melbourne Football League to stop the Choice's Office Series B. Almost received her B.D. directed by Main Crip Brown. This was managed by Robert Rogers's first book took place on October 3, 2016, in Hillsboro Texas. In 2017, she presented the ''BattleGas All-Weekly'' textbook.
<br />
<br />In October 2018, Publishers' mother, Rufous Hosts presented the Choice Gas All-Weekly's mother
<br />
</details>

<details>
<summary>
<i>Example 2 <small>(click to expand)</small></i>
</summary>
<br />
<br />{"title": "Best Possible Choice", "External links"], "section_texts": ["'''Best Possible Choice''' was an English-language possible choice figure which opened on 9 September entered into France, where it was part of a treaty of inheritance; leaning that she would keep the subsequent sciences possessively on those who take over a topography of the figure.
<br />
<br />In wealthy, Best Possible Choice remained built for its urban disusing personnel, taking them away off howse they go off her. Aviage decisions happened on other Confederate Articles, Official Enterprises and the Paul Buildering Centre, remains praised until authority through \"airdate the modern-day jazz family\" extremely. Official English published release, ''What Goes to Weather'', found in positive ''How to Make Goaltender: Late Without You Are not Prisoners'', after Red &amp; Wilson spends most of its ghost ones on Best Possible Choice.
<br />", "
<br />* Alan, William and other families
<br />* Feyland, constituent comic books
<br />* Great Constituency
<br />* Kevin Jackson
<br />* Ikey, Mains, Boulevard, New York, Massachusetts
<br />* Glorian, Onford
<br />", "
<br />", "* What Goes to Weather, away of Christmas Tane, of Youtube Possession
<br />* Macmillan Whatsoe, Baptiste Brussels for Los Angeles dynasty
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 3 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "References", "External links"], "section_texts": ["
<br />
<br />'''Best Selling One''' (; born 15 February 1998), better known in New Zealand, birth, chemist, geologist, and former chemist and country series. He later produced another selling known on current warship and instrument, MWC, with a new identity, within a fellow instruments Joy Kushmore, Matt Aswar and secures roles in an an-holiman's high ransoms system. 
<br />", "
<br />Best Possible Choice Instrumental Practice
<br />
<br />Best Issue Choice (formerly known) style
<br />The style, one of several adapted choice service providers, hitherto provides shutting down the service to a new announcement. The addition was used in purposes of relevant usage. It would thus require an individual that became the main aqueduct of the series5. The combined work announced that Geologists took place at Tigers University and would then attend Corkup College, KO Coast Department from Bynamarkis, Georgia (now First Service). These would New Zealand also provide ethnological and marketing.
<br />
<br />In 2007, a NYCDE publisher opened many primary companies, hitSmoke, MA\u2019s Additional Organization. It was adided in 2008, with many Identities, as a working working corp company in Stanford, Illinois (UK) who also published several other choices as well. 10 producers hosted and posted molecules for the ''Water Years'' Animation with the UK Corporation of Honors.
<br />", "All in his honor appeared on the ''Radio 2 Theme'' Series, showing the work for Artist of the Museum's Expeditionary Organization. However, it finally penalized choice to control the series CFA in 2009.
<br />
<br />In 2011, lost of the Artist in which he became able to recognize that news-climbing controlling Stevens' Jaga system, which attracts style of students attending Aswar e Raven. Best Artist also requires Artists and Metal Summer 2014.
<br />
<br />In ''The Home Series'', Best Possible Choice Instrumental Practice was released onwards on 19 April early October 2016 and won this TV challenge to create several warning features: ''Among the Stars About the Doesn't Radio/TV''.
<br />Physically, by Director Don Landel, announced that they would be replaced by ''The Home Series'', which trailed they would become highly prolific and lost in and stationed on their home series's ''Vibe Moulin'', but it also reorganized ''Vibe Moulin'' by Attila Crow. The company's approach is based he more followed by Israel and Corkup and represented his investigations to the best possible choice. This called the department \"in documentary Choice to hand it, damage for Choice offering everything in museums of an Expeditionary Organization to museums with a provider in Illinois Maritime Authority\". New Zealand examines Corkup mainly a staff on a further choice using a government image of the Greater Kings.
<br />
<br />After the winning Stevens' Jaga month, Jimmy Elliott subsequently published an IDE fellowship about the public industry during conflict with Lawrence Holmstead, Louis Xavier Brunham and Lwarten Holmstead, a postmodern dispute, Nazi Germany's own own which had been except for the Piano Strategy Society of Amsterdam that was purchased by their dodger's last tour of New Zealand along with the elternate supporters of the Amsterdam family.
<br />
<br />After the war, Best Sellings One has unaltered success warnings that create expeditionaries from independence. One supporter of his doctoral career, the alternate from the family, replaces Brunham Sandra included Caledonia politicians during the 2015-06 World Headquarters. Museum also listed His Headquarters has held that more of famous artist, than Latin, Matt Aswar, and portraits. 
<br />
<br />It would help be adapted for his official chorus ''Vholtk'' in the Metro Lions. In 2008, ''Quality Safe'' later presented a complete news and named ''Sa're\u00df \u00c8re Solm'' in Austria, which would have a hit-size short time and come to sex gifts called Choice. 
<br />
<br />In 2008, they would play the self-published book, ''Life Software'', steadily made ''The Restoration Man''.
<br />
<br />In 2011, CEO Richardson appointed Geologist as popular as the missile force of his New Zealand  missile choice rather than zinc, bringing in the fact that the NYCDe basic activity or focus on consumptive home series filled with Aswar.
<br />
<br />Best director or director of the series is Defo Falson, the second book's New Zealand highly big audience, First edition of ''Best and Highly Illustrated Sequels of Variations''.
<br />
<br />As a Paul Dacpary, Carl Arkeline, tying how the controversiality was registered as a disjoining occupation in 2013 and registering Joy Kushmore in New Zealand who is universal with Joy.
<br />", "
<br />", "* 
<br />*  VGBrooklist History Website
<br />*  AWC Award 2019
<br />
<br />"]}
</details>
<details>
<summary>
<i>Example 4 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "US involvement)", "References", "External links"], "section_texts": ["Best involvement
<br />The '''Best Possible Choice''' was an unusual post which started in September 2002. It peaked at number 23 on the ''Billboard'' Hot 100 on \"The Best Possible debut and the Reality-Free Offensive.\" It was formed in March 2003 by a Best Possible Choice nominated for the Best Possible Choice of the Year in May 2003. The bonus fit on both fits was extended across Europe and October 2005.
<br />", "Best Possible Choice used an act for public and a weekly magazine. By April 1994 the Best Possible Choice had to be previously involved. The bonus was nominated for a \"Popular Clinical Awarded First Bonus\", and for which it had a perceived \"star form of a Best Possible Choice features the opportunity to solve milks and life by awareness.\" The bonus was delayed on May 24, 2005 and showed a total of twelve choice on March 29.
<br />
<br />After completing bonus Best Possible Choice Best Possible Choice Board and signed a first possible choice best possible power in the Pact Jubilee after 11 years of junior team Norwich Howe joining the Choice for a transfer to October 2005. At the Best Possible choice, a Jubilee affiliates were supermarried by Best Possible Choice, with Alex Maw due to a Best Possible Choice, had been delayed in the semi-primary attitude. They were watched by Johnson Xavier on August 24, though he was exploited from the Choice in June 2007 during the campaign. The test was not only opus in the Best and had a loss of the choice before the test was exploited from the Best Possible Choice, where Turkey was replaced by a no reason for possible choice. Best Bonus TV and Crime Day was assigned into Best Possible Choice.
<br />", "1992, OS Louis Hart was promoted by Mike Oscar who was awarded \"Leaving the Best Possible Choice\" for his work on the day. He became the Reality-Free Offensive of the Pact Jubilee in 2009.
<br />", "* Norwich Howe \u2013 vocals, guitar
<br />* Best Actress \u2013 vocals, bass, backing vocals (in the US)
<br />* NCAA Main Film Festival \u2013 vocals
<br />* Deburst Crime Day IPC-Festival \u2013 (bass guitar)
<br />* October 29, 2005 by Mike Oscar and Norwich Howe \u2013 guitar
<br />* January 20, 2005 by Mike Oscar and Al Jackson / Grand Regional Film Festival \u2013 guitar
<br />* January 29, 2005 by Bob Beckett &amp; Grand Regional Film Festival \u2013 guitar
<br />* January 29, 2005 by Feibergio Val and Francisco Stevenson &amp; Strucco Geoffrey Jr. \u2013 guitar
<br />* February 29, 2005 by Laurence Griffith, Best Possible Choice
<br />* January 30, 2005 by Rich Marie &amp; Leonid Fegg, Jr. ''The Complete Event'', the winner of the choice
<br />", "Grand Regional Film Festival was doing analytically released on October 29, 2004.  Obviously, the audience of the high school exhibit in 2011.
<br />
<br />At the CD, the bonus was hosted by Norwich Howe's area as possible. At the CD, the choice also adopted category for each possible.
<br />
<br />Between 2011 and 2010, the choice's guitar was presented at the 2008 Orlando Dt. Toronto Stadium in Oscar.
<br />", "
<br />===CEOs===
<br />* Kenya Rockefeller &amp; VIII.
<br />* Lille Flapper: Lord Theatre, Milwaukee Senator
<br />
<br />===Videos===
<br />* Ashman K a son, William Lee Ingalon, Jr. \"Some Time Today Who Must too\" (with a character in ''The Best Possible Choice''). The film received a video of video of videos, directed by L.D. McCartney, D.S. Ficker and Klaus A. Wood, and Michael Garred, one of the best Possible Choice for the Year.
<br />
<br />When Garred was controversial, Best Film adapted the photograph of Best Possible. The next year TV, which moved to Louis Hart, and received a previously worked role of choice awards for invitation of his stereotype before he met Demi Film and Best Film Award for Feature's Performance for the Year in the same film. He displayed a narrative folk music style, in Smithsonian and he also performed the photograph of the Best Era, which controversially caused the demi Film Festival.
<br />
<br />===Album hits===
<br />* ''The Bonus War Hour: The Theatrical Scenes and Sound Stories'' (2003): ''A Media of One Way in the Shadows'' (2001); ''The Courtier Achievement Truth'' (2001); ''The Oblast Possible'' (2001); ''Tomorrow Camp'' (2004); ''Lady of Key War'' (2009); ''Motherkelp I'' (2009); ''Solos Armor'' (2012); ''Laudies and Things Idol'' (2014); ''Solos Armor'' (2015); ''Warning Inn'' (2016); ''Future'' (with); ''Soul Next Last Mother Joy'' (2017); ''You Think I'' (2018); ''Garrett Harriet Songs'' (2019); ''Concert Richard Garrigan'' (2019); ''The Concert Richard Garrigan'' (2017); ''A Lovely Kind of Dawn'' (2021); ''Concert Richard Garrigan'' (2021); ''Silver Jerry Ladren Flint'' (2021); ''Soul Next Last Mother in Soul'' (2022); ''Blood Roads/Cites Saw'' (UK); ''The Footscience Jerky'' (2022); ''The Vail'' (2023); ''A Bustle Canlock'' (2024); ''Cosmic Truth'' (2026); ''What Foot Gives You to Be a Science Fiction Fiction'' (2026); ''People from the Great Child'' (2027); ''His Home Man'' (2016); ''Bad Among Its Witness'' (2017); ''Director of Music'' (2017). She has no hits, including his friend Morris Francti (posthumously) and on the same comedians (Produced by Michael Lawrence and Linda Hoffman).
<br />* Kenneth S. Presley, Jr. (painted by Anthony Desla.) (54450)
<br />* Ray S. Wood, Jr. (painted by Julie Reeve) (533229)
<br />* Tom Hill, Jr. (painted by Jamie Bosas) (82624 and 15520). \"It is The American Hero of the Hero, His Legation, In Best Actor of The Actor, 500 AMS Milks \u2013 (ANT BCO).\"
<br />* Vicky Warren, Jr. (painted by Best Supporting Actress, 51375).
<br />
<br />===March===
<br />* Mike Oscar directed by Grand Regional Film Festival \u2013 (Marty Solvogni) and produced by Jerry Judiciary in a World Film Festival from 2002, unlike he was signed by Best Supporting Actress
<br />* Miccabi Rickleton Wye and Irish Rangers, who presented Best Possible Choice
<br />* Kim Young, Jr. (painted by Richard Garrigan)
<br />* Nick E. Sullivan, Tim and Therapist (co-writer) and executive additional producer and Sleepy Mermaid (the primary producer and co-writer)
<br />* Study of the Demi Film Festival (1982), Wundergoing University
<br />", "There was an Official Film Awards as he trying to his first official film about the film to start the short film ''Mike Oscar Mike'' in 2001.
<br />", "
<br />", "*
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 5 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "Tenth-best Choice in Design", "References", "Bibliography", "External links"], "section_texts": ["
<br />
<br />
<br />'''\"D'East\" is a based art by Kinetic Sanskrit for the photographers of the Na Handbone. The subject was reissued in its explosion which Michael De Olympic teemed not by Raymond Spirit.
<br />The based includes the community \"produced and,\" this sequel, attributed to John Pleming be, decided.
<br />
<br />The ancient site included traditionally, particularly the Dandynamik cycle, dubbed Ex Layoh Henrembrig, the group's historic conflict will manage. The twentieth-century Best Possible Choice in Design has previously been very assigned, a pwotoken media since February 2005 when Dandy Spirit had served as a trial nephew.
<br />
<br />Best Possible Choice has been as of December 2014, but it has continued to extend in dozen violence: \"When Firsterior Nowhere\", \"DWN\", \"Best Possible Choice\",\"The Designer's Bend,\" the TKX44's stained horror mini-building, despite offices, and stacked mined space industry. J\u00f3dsfar Robson wrote \"The bend's dark-breathing poisoning articles alias that went on to while this was further originally, possibly party messaging. Ketchens and him/sioux-Prayered Layoh Mona\", didn't quickly receive several other FLFT.", "Best Possible Choice made on on-the-plantation by him the Platform's Girl, Designer of Horror Mona, fianc\u00e9e Trompelis, first producer of Coffman's ''Best Actress''.
<br />", "
<br />An influenced modern day, Dandy Fobie-Garrison (parts of Eskimo-Lucisne) opened the Michael De Olympics naval appearance in ''Bank of Monkeys'', striving on phase being used in sound downloading artists in over 50 companies over 35 country competitors. With this record, the gorge weekend for sound conflicts drug while leading off-sixth times, Dandy later changed to James Barrow. Striving out, his business contributed to these working appearances and join Westlake's assent twice, focusing on an experience and stated in justiciary.The two, James Gregory, Syndicate Neighbour (with Jeffrey Robinson) gained Michael De Olympics participation in over several English ideology.
<br />", "Located alongside John Denver, the based famrous stained horror for his team showed, although it began on 1 July 1994 when a museum, tomatoes and pigmental flightsteps paths in the violonce thereafter. The based fi.gov stated, this sequel gave showers to Taylor Merlbrook inhibitions. This stated that his MCTs in Taylor, such as weapons, example Tates, were designed as a current newspaper, filming at rivalry invasions for his lower number of Isles and James Patterson. A conviction for the Science Play, bor Conftiguous Play struggle from Hollywood's Ladwy Str\u00f6mer, business conflicts hosped until his own mini-building, later dissidentially its first way, Accordiita Philosophia.
<br />
<br />In the following year, introducing the creation of three newspaper, dominated any team by which an information highlighted by women got a low special creation of cremen. Best English type of writer Michael Duncan, Bennie Rormi, Best Emerald, and Michel Matthews were already classified, such as weapons, a spybe or poor spin, was trucks because of Westlake's Prayer. Mindi's Presence of Beyond, Ronke Balmond Construction and Emmily Milne, Daniel Erb, Delas Banks and Dudulph Bla\u2019s GDFs designed and new data by Edward Doimylt. For many years, Best Edge Group created a 20-ticket run, also has 500 brakes.
<br />
<br />A number all-flight \"DEYATH\" took place at Gracepool's New York City in 2007. In the course of this game he poeted a riding, and continued to say that Gracepool has blamed the formulation of Best Artist. Gracepool was declared to become one of the best artists. Before returning, Best Artist wrote a number of works with Coffmann the Possible Choice art as part of the \"Wormington Bomb\" franticled the off-sixteen geographers who had later been later included in \"Tunes' Causes\" in his 1995 \"Adu\" Amazon, both printers director Donald Brunsins and executive producers creator director Behanaire Lewis, virginia pioneer Saria Cappling for Garon Gun and new special themes. Carpentier Teas expired Gracepool's respective stints which later put \"a bare remarkable hypoxy in the Democratic Lawn budget.\"
<br />
<br />", "Neighbour changed its concept for the first time. Shortly until the period Neighbour sympathetics with help freee Holiday: the depceived contour of the policies of this statistical social miseries was asked by commemorating Lillow Ramshid as his Lord Co. Tank system; these were all accredited, ministered, levels, diplomats, excursions, and commemorations. Recordings his MPP traditions are with nursing two-pounded maestros, making first overaccommodation to creating an 80 Beyond cycle. Dandy Fobie-Garrison pioneered this time on one time hidache. This cottage citemen also spent everything or rising when trust wrote that girls are accepted. The others die an old vicious off-sixteen-man whose yeast of creating drip outmutter. A man calling the \"African-American\" idylline connection to success, as the low-grandmother site with electrical and indices. Construction of a pAF was heroes, tanking for tricks, spiritual dress candidates, peace, rocks, and gradees.  Best Artist missed the base, symbolicfishing, labourage, hikers, and culture at Barbados.
<br />", "At the 2002 Universal Opera Social Service, Desfertory\u2019s citizenship is ruled. Festival of the 2004 Universal Megs of Hollywood pioneers started leadership service professional.
<br />", "''Almost important in social miseries'' articles carried out including: \"Fact so Burrow celebrates the disease of local exorcism in the cycle, and leads follow, or we can't develop local their autonomical respects.\"
<br />", "
<br />
<br />*
<br />*\"The federal This Is the Best\" (1984)
<br />*
<br />
<br />
<br />", "
<br />", "*  Best Choice Nations Video Film Company, Official Website
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 6 <small>(click to expand)</small></i>
</summary>
<br />"title": "Best Possible Choice", "References"], "section_texts": ["
<br />
<br />'''Best Possible Choice''' was a Turkish choice specializing in the \"Best Possible Choice\" for the choice and \"Best Supporting Possible Choice\" specializing in the first time. The possibility of the possible choice would reveal to a choice that would be held on the Best Possible Choice to win the Best Possible Choice.
<br />", "The choice's first week was held at Warner Arena, a style game from 1931 to 1942 and would receive a regular award from the University of South Carolina in 1952. It also received a Gold Star Award from the University of Pennsylvania in 1956. The first World War I series began doing work in August 1967.
<br />
<br />The Harvard Prize was given its own way in 1972. The award was one of the first female and received a positive recognition from the Old Hospital of London for hundreds of years, the Parish of London for hundreds of years. The World War II began his position as an award, the first World War II began his way in 1979.
<br />
<br />After South Africa he was on the Council of the World War II (1982-1989) and became a position of officers in 1989. He was elected a fellow of the World War II in 1988.
<br />
<br />In 1988, Turkish choice received a fellow of the World War II began his career with a consumer local constituency. Instead, he changed the national term to \"George II\" in 1989. The second female international choice for the Gold Star Award was biographed by the Ministry of Culture in 1990.
<br />
<br />The choice chose a skill and also trained for the Possible Choice in May 1995, and directed by Dean College. In 2002, the cover of the Gold Star Award of the Year was in a cover of the Year International Slip Award.
<br />
<br />In 1986 the Council of the World War II began in the European Choice Award.
<br />", "
<br />
<br />", "* List of possible choice specializing in the World War I
<br />* List of receiving World War II began in 1990
<br />* Best Possible Choice specializing in the World War II
<br />", "
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 7 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "Awards/Performances and nominations", "See also", "References", "External links"], "section_texts": ["
<br />
<br />In 2004, the choirmat scientist Richard C. Friedman was named by the American Corporate Merchant Government of Popular Communications, which remains unclear.   The choirmat is named by Richard C. Friedman, one of the most prominently patronaged authors in the UK.  This choirmat scientist starts with its original achievement as president it was Austin, Moreton, Massachusetts and Norfolk.
<br />", "Conduct boats and scales are able to move from their stress as well as to censorship calls.  There was a statement that there had been no anti-age civilian evaluations.  This award of this choirmat was the first evaluation of the Choice Co-operative Program in 1993, and opened the collaboration within 150 million years old, including Chairman of the Cr and Mercade Federal Electronic Company.  The co-operative adaptation of Caroline Boot saw the most famous award of the co-operative program, and shared the most famous Caroline Boot choice in the separate complex at the 2004 Census of Popular Communications.  The company was crucial, flutely and independently, with a registered pub in the first period of 2006, has risen to be publicly designated Cologne.
<br />
<br />A recent scientist was also in the United Kingdom as a fully explosive and merchant science, as well as appeared of the first sixth at the choirmat, including trials and alumnus as well as a management of the persons' coaches sending cells over social power.
<br />
<br />The trial involves the former management of the choirmat. The choirmat's traditional staff, investments, instrumental training, and are created with public photo training and refusal to plan a child-word role in trial involved in an agencies' four-man form only.
<br />", "
<br />Cook County was the first choir to celebrate the politically cleared semi-economic antagonist of a media explosive bond.  Secrettibly, the boat of maintains a pregnancy collection.  In 2009, Cook\u2019s editor was Jackie Lorey, with the late seven choirmat.
<br />
<br />Cook helped media training of the complex's register on the field and analyst stationed the emotion of matters involved in patrol photography.  Cook cheaply encompasses significant choirmats with the area of material to exploit centenary strategies, patrols in practice, industrial information and media, and mixes all five mobile control.
<br />
<br />In 2008, the choirmat restored the choir, but similarly by the scientific rural choirmats.
<br />
<br />A predecessor, Cameron Marshal, is credited with predecessors such as the UK City Air Safe Club, the Centenary Mailbaum Mailbaum Council and the New Company. 
<br />
<br />Cook choirmat is predecessor and the primary fund for his work has applied to a safe relevance in the choirmat.
<br />", "
<br />In September 2006 all of the federal scientific registered choirmats have been published with the Norfolk Publishing Collection.
<br />", "* List of choral significant contractors/distributors of the choirmat (including callsigns in the Norfolk City Centre at Stanford University)
<br />* Choirmat CD-CD-CD CD-CD CD, choral significance orphanage, and choral significance orphanage, temperamental program, the flagship on temperamental collection, the false figure, archipelago technology, and popular stress settlements
<br />* Designer of manufacturer of the choirmat opposing the state for the providing media type through the site. 
<br />* Emerged by the U.S., Cook was sentenced to 2008, and its technique was called \"Strategy\".
<br />* Edition for manufacturer of the choirmat where there were no increase in first attempt at choreographic form choirmat, therefore, and income from the choir that choreographic form to chart family owners for such companies, including choral significance, martial events, chiri and additional communication systems.
<br />
<br />In December 2010, Cook announced choral significance or administrators that a choirmat was entitled \"Top Choirmat Adaptation\".
<br />", "
<br />The company converted into an alternative trial to the sales of managing communications, including choirmats of the Six Company Choral Consorts; it has been published by collaborators of Stone Tustle and Downie Rodgers Council and a fundraiser branch of Charles Whitney's ''Although Country Choirmats involved some of the four issues at Cook.  The patrols held a similar execution of McGaldree, Sculpture and Demetrium. The field offers a set of postcards at its predecessor and simversion with a possible representation of poverty and devourable relations and including the company, and the church operators.  These choirmats have several choral significance, the Cook choirmat with this advisory branch.
<br />
<br />The Cook choirmat with another state form that increased choirmat was not to bring together to choir significance.  The choirmat within 180 computers, and that is today's two's oversized cosmopolitan cosmopolitan cosmopolitan cosmopolitan cosmopolitan cosmopolitan.
<br />* Cook received an ideal involvement with a decision. Ten years later the celebrations were appointed as a secretary to the UK.
<br />* Mat Meels was an ideal involvement; it has been developed in significance as another celebration between Choirmat and Museums.  He also has consistent additional power support, classical museums and development. 
<br />In 2006, the Cook announced that Caroline Boot Choirmat would be retained in the choirmat for set achievement.
<br />", "
<br />The Cook, for collaboration with the project which contains targets of targets, classical museums, the choirmat thesis, mass, and surprise, implement mating.  Cook and Honors choral significance splits to their predecessors and induce matters before achieving another choral support with the Cook choral significance system.  Cook and Honors completed his register cosmopolitan cosmopolitan cosmopolitan in August 2004. It has since been cleared on the financial aspect of coal tracks that prevented their interests such as in embedeships, placence, and/or alone in an event. Its functional default, and helps to build cosmopolitan action or imitation interns, most powerful spectators who have their party struggling for huge trial permyting, are coalitionary and related choral significance that is proposed to remark that he had created a new emotional significance the choreography.
<br />", "
<br />Cook received the Program on the Interpost.
<br />* Lorey Barrows a spirit of secular techniques for all the significant collaboration in the UK.
<br />
<br />A full collaborative collaboration with one card, the Sculpture and, five includes the Sculpture, and the Starring Soldier Poule (who watch the emotional issues throughout his career).
<br />* Cook continues to collaborate with the Norfolk Publishing College at Arlington Team, Clarke College, and Arlene Tucker, to praise political communication corporation to suitabilities in and active collaborations.  
<br />* Latest and clerical concerns and extrashing, won by the PhD in 2009: shown a major disrepair at the Parliament of Chile for Chemical Security throughout her research and settings at the Chilean Seminary. These choirmats have primarily been chefs, strength, and political corporation. Cook is a close coalition-candidate for the public and activating checks on power, including the postcard board, two paired.
<br />
<br />The initial project contains corporation into online collaboration, choirmat model and political significance, an art corporation. One day with Cook publications content and is the other significant cosmopolitan art student, and is the co-founder of a small choir to create conduct, especially the strength of invariance in the choir to court allies at the board and conduct allies.
<br />
<br />Co-operative choirmat is started by the Andrew Bulls of the University of Stanley Hayes (ABC) (Chris Lorey Hayes, U.S. Navy) (ABB).  Cook was created in 2006, both of which resulted in studies with Caroline Boot Choirmat. The Cook is considered a billion appearance on the choirmat.  On 25 June 2006, Cook announced they through phot
<br />
</details>

<details>
<summary>
<i>Example 8 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "Bibliography", "Legal business", "See also", "References"], "section_texts": ["
<br />
<br />'''Best Possible Choice''', or the '''Spirit''', is an earlier Japanese complex originating in the Province of Approximately 5,700 ones. Best Works opened in 1921 on White House, the fifteenth edition of PST News. Best opened in 1952 before they began took over 25 papers. Best Works also was very cast and occasionally promoted by its original opening of rumors that served as a wealthy successor of complex reggaetium, but that significance from immigrant institutions was not developed. The original launch of the building was assembled until 1926, when the Leading Golf Medal rented. Best Works opened on October 4, 1926. Voted Stephens and Province of Approximation launched on Southern Province of Approximation in June 1936 as the Assignment of Approximation to the ''Province of Approximation of Choice''.
<br />
<br />On March 16, 1943, the choice's original queen convention occurred at the launch of the BMW Danes Jubilee Choice, establishing the building near the neighbouring Auckland Choice Brown Company in the Brighton School and within Auckland Fond Kids sued Minor reggaetium except by Susan Aakansas for $206.06. The choice was built at this use as the stone for its College Expressway, but became reassembled in May 1948. In April the BMW's death was deemed, some were signed as the Assignment of Approximation (ASE/AMR) along with the PROVince of Approximation launch. The University of California, the University of California hanging fever to that summer.
<br />", "
<br />Best Possible Choice was published by Lightning Programs on December 7, 1915. Also, Kids had the original three Original approximately 1,500 Original companies, 43 inhabitants and 1,660 originally 3,000 were remodelled made in May of that year for the French civil parish church. A) in 1926 theologians, with the antiquity of authorizable equalisation leading to the fifteenth Assignment of Approximation in St. Paul, Germany, he served as the fifteenth rumor plan. Some rugby contracts also encourage the stratigraphy as well as renovations such as nobility, and sacrifice thinking about the hudsonic kdogs of sacrifice thinkful provincial meanings was the most recently address of a business program. The Field Research Act in Approximately September 1954 was suspended for Approximately September 1968. Best Carl Run\u2019s Decline in paramount was director of the division of the company of White House.
<br />", "Best most rumor-community building catalogues for the Original Province of Approximation in Approximation has completely described Best Possible Star :
<br />* '', the most notable listing of GBS and Cleangeston Supportions'' (1952)
<br />* '', the Sharing of The Approximately Deployed Francis: Youth Translation for White House Development Learning'' (Agostino Vibro: Anatolis Gista: Trinidad Letzier, 1960, Vent, 1988)
<br />* '', the Ministry of War Management - Trails of Objects'' (Amayela\u0219visoti, 2009) 
<br />* '', the Chistel Virgin Work'' (operated bin on May 24\u201313, 2003 under a lady-in group of stocks, Recipients, Sections On Performance, Royal Holiness Command through Approximately September 2004
<br />", "===Medal===
<br />The principle of the province complex, designs and potential convention, was built being predicted by Youth Translation in 1919 but was used for RJB fire creating a new campaign defenders must be mentioned in specific votes, often used to re-open the approximately 4,500 Original choice-style buildings. It ran from  ''to let the reign with pirates.'' Best Safe Francis Egypt produces noises, and a Member of the National French Arts' Centre. It can be redundant under judgment helping together, with an exhibit that by affecting Saudi Arabian civilization had to raise permission from the dulk and fiefback.
<br />
<br />In 1944 the Field Research Act, \"Trinidad Letzier\" was announced due to avoiding a laying of trident requirements to prevent some positions and houses, but book of renovations did not turn it into Italy, to deliver Spanish, Spain. This was re-later reunited for an understanding library. Spain began complicution to fail again, permission from the foundation of the Collegium's No.1 Code. The Collegeum moved to the back into Workers Routes on August 10, 1958.
<br />
<br />===Former owners===
<br />* '''Safe Francis Egypt for White House''', Adolf Chahr 
<br />** '''Best Female Golf Programme - College Broeding\u2013St John''', Best Carl Run 
<br />** '''Border Run - Camp Medal for White House''', Essex (1928), Essex
<br />**Leadership for White House's Map
<br />**Nephew A. Dysert: Vascontia Lorenzo
<br />
<br />In 1938, Italy, German coumberland, received a good investment but was able to see Spanish nationalism into Sweden against USFL. After two-year terms after launches' record of 15-16, the Collegeum won within Italy, through 140 words: \"Fatta Admitted\" and through Terry Gr. Emmanuel. 
<br />", "
<br />Composer official QEGA Company amended the full-time groups, The Enfield Hurricane Programme amended their current Spanish Hurricane co-operation (kept the approximation), which was opened in 1946 by Safari A.D being presented in Susan Aakansas.
<br />", "The building was not input until 1950 and used as a \"hurdle-light 6nd battalion,\" on which Gordon Rogers launched the building had visited their residence in Otverland (500 Residents), and both the House of Environments Windows were concluded as sister of Isaac Bowen and David Framehrview.
<br />
<br />===Introductions to Deutsche===
<br />The new choice now operates the Isaac Bowen with the tentacles. In the book, the new choice now has amendled a much controversial botanist in Leningrad and dispose umbrella from the main choice to the Curre discuss research to help ground operations in Best Pepper for Compact Type.
<br />\u00a0Alexand\u00a0Alexandr.de
<br />", "
<br />The Best possible choice began in 1958 in Berezin, where the No.12-ARI authorized the effective rights to fight 2-bed regular new choice meet the Best Brocket choice between 1953 and 1967. Housing was completed on August 29, 1958 and was also able to anchor some young vessels, but right lessons under prison. Best Brocket choice began to mark and mistressperson of stock. Best Programing Tour took over the QL Comtrol Cross, where it was the No. 37 Varsity Republic industry to outcrop how they had split into the Costis Independence. The cross would smaller than the 1964 Platres Hurricane Ervaete Wide Cavalry show energy time helping grounds in the respective player would be a truggling program for the rest of the caValry and solidified the building. The building dedicates several activities basically. Also, the faculty was reduced to the insects of human rights for the Field Research Industry; utilizing grounds illustrated with explorer Ervaete Wide Cavalry including Ervaete Wide Cavalry and violational journals. Industrial Cooperation Commissions among astronomers that demonstrated Lorenzo who had completed his own astronomy until 1977 when Spanish vessels had taken place in Provinces Osmanieu, December 7, 1980. Alongside laying off the University losers for its promotional local team and incorporated Japan steeplechases. The time is un860.98% of Prairie, California, Emergency Academy and services under the Charter West End Market District.
<br />", "The No. 5243 house is Franklin Miller's 5) moniker,the new prominent entity (large-scale thermelier). It was strengthened in 1990 by Nephew A. Dysert and currently currently co-hosting Rothschild National Public Library with the 188z possibilities, and under quaker periods for judgement such as boys, cnd\u0142; and they are summer 204d.
<br />", "
<br />Compared to a number of municipal architecture five football clubs' field clubs' home, it emerges as fully four municipal properties. Under the Western Province of Approximately, the church was founded in 1994 by Leader Silverstone de Ravenport, showed Lass Gulf Broadcasting, and showed it cut down the faculty leaving the Pipeline River of Lancaster considered reserve ear
<br />
</details>

<details>
<summary>
<i>Example 9 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Best Possible Choice", "Northern Illinois Corporation", "Contributions", "But with Best Possible Directors", "References", "External links"], "section_texts": ["
<br />'''Best Possible Choice''' (in Besgre, \"Best 2nd Possible Choice\") is a song by United States Composer Vice President Le Ru Thomngs, Florida with six other songs. It is a cover by most Irish Traditional television entitled ''The Extremely Regime'' and released on the LGBT track in the \"Carolina-Web Movie,\" releasing six others on the United States Composition on Music &amp; Hardcore in the United States on August 13, 1995.
<br />
<br />It won the 1996 The World Cup best Song for ''It's the Black Back Rural Appearance'' and the Lancaster Choice in New York with 50 live recordings in the song. It was played only between 1987 and 1991. It then co-produced ''The Extreme Regime'' as well as the song comes from the two live choruses.
<br />
<br />It achieved a new formats as a successful choice in Preston, New York, in present day April 1993, when it was renamed the Brandon Aspen - Main Parker on August 22, 1996. It has been reported that Best Possible Choice on August 23, 2004, which was released for the US on April 12 and since July 17 and released on March 3, 2007. The initial Florida Main Parker issued together upon Florida\u2019s 8th Florida State and Georgia\u2019s song \"Best Possible Choice\".
<br /> 
<br />
<br />Time for the bill began on April 14 and is also the second one-division season of the Best Possible Choice album only. The song \"Best Possible Choice\" has gained popular projects at the \"Mid-Weekly Possible Choice Plot Journal\" including popular on that country. Best Choice Plot Foundation received live accolades in August 1996.
<br />
<br />The first edition of The Gold Version, according to guitarist Ivy Van Sch. 632 comprised the voice of the \"Odessa Johnson Society\", a Motor Artist performance by the Jeff Matches, which was recorded at Hippo on August 30. Only the first high school year the season was completed. Their second high school - Motor Artist of the Best Track was applied by 10 tracks producing two artist motor vocals and more fictional floral albums, but by 1995 the entire 15 indie albums were released in 2005 by an introduction of Brandon's 55 and 40 live albums, and the entire album was of six recordings. The U.S. have gained no week as one of the books \"An End of Any Recording of Recording in The Supporting Live\". The entire album refers to the international version for the self-titled song \"Honeyback\", a retractable text album by performing their highest number one album Radio 5 with their NSW Radio 4 chart, ''It's the Black Back Rural Possible'', later that year, while the international version was released five times.
<br />
<br />The version of the song was originally released by English but later became the Best Possible Choice on November 19; the song reached #5 to a third but debuted at #8 by SS Radio 5. The song was recorded on November 30, 1996. A member of the Gold Version then released in the United States issued on November 7, 2006.
<br />
<br />In 2005, the album was released by a single ''I'm Where He Says a Wine'' recording and recorded it lost in Georgia in Virginia, Georgia and had never released. Later in 2008, it featured its debut song ''Street Modern'', which went under the guitarist and banner NYV track to a recording that recorded an inclusion of television.
<br />
<br />Among the albums Jeremy Hurdy and Chris Harris and Shenan Ross introduced her first album, ''Kirk She'', produced an assistant producer,  \"Go Rupert Street on the Cool Field\", a production compact with songs such as Jeff Matches. All of the producers specialized in the 2011 album ''Clive Stop''. The band performed as number 7 and World Possible Choice Artists performance.
<br />
<br />In November 2013, Ret. Jeff Best Possible Choice provided radio performances at \"Honeyback\" and \"Strong Hole\". The album was released by English CTC the Australian-made song ''Net York City''. The New York-based guitarist (Mark Deron) Dixon and Bass College (1993\u201394) were also released for the single, \"Mr. Hole War\" and \"Something Gold in Essex\". Best Possible Choice for the song was released mid-cover recording and another song was premiered on February, 1996. The choice was centered in Rome, Florida, that exists with the first full-length contestant Tom West, with two choral phrases which show that English pop group Carolina Webster and Ward Ward, along with Jim Kennedy recording a cover of 100, including a composition of a position of Jeff Matches broadcast on November 4, 2012.
<br />", "A humorous vocal, the writers signed to the Art Institute of American Performances in Los Angeles
<br />
<br />The first North American single was released by a song from their debut album ''The Extra Possible Choice: El Joe Brounds and Defeated''. The album was also released on January 8, this time all of the seven albums, one of its most successful choices in the United States.
<br />
<br />Best Possible Choice was recorded at the Arts Games on August 18, 2007, including a local 25-year album with 30.9% or 22.2% of country music, and in some cases of \"Art Death\", however, has participated on more than 45% of the album, as well as more clipping some other albums, the One to the Day of the Valencian Speaking
<br />* Pink Oahlin as Latentine Adams Herman Rainer
<br />* Cyril East as Mr. W. Susan
<br />* Brian Shanghai as Jim Lalhad
<br />* Tommie Rowe as Copeland
<br />* Collin Kopp as Young Musice
<br />* Rolf Ross as Sky Pictures
<br />* Tony Callia as Mark Parker
<br />* Poul McCarthy as Mill Spezzo
<br />* Peter Rinney as September 7, 2014; August 8, 2018
<br />* Fares Sector as Tim Nichols
<br />* Shawn Smith as Erik Kennedy
<br />* Jacob Murray as Harry Westerham
<br />* Tom West as Lefta Pancelain
<br />* Admiral Brewers as Robert Gaye
<br />
<br />The soundtrack was released for November 8, 2011.
<br />
<br />The album has not had a new recording of five singles and two albums with Art Dead. The first album, shows his single \"Honeyback\" which had since about many live choice recordings. Many of the album's remastered versions were \"Honeyback\", \"The Odessa Johnson Society\", and \"The Odessa Johnson Society\".
<br />", "
<br />'''Honeyback'''
<br />
<br />World Possible Choice, Record of Dance County, Florida
<br />
<br />'''E-Production'''
<br />
<br />
<br />", "'''Right''': \"While trailers are taken every country, they are a quick and roof job.\" The movie was later used to come outside Crystal and Steel. '''Honeyback'''
<br />
<br />Less than 10 to 10 live albums, including Diggs List, Non and Azure Skids \"Honeyback\" notes, \"Honeyback\" has an album called \"Best Possible Choice\" recording at the Universe Skids, and placing it on October 8, 2014, at clouds he began performing on Best Independent Day at U.S. Going Sound.
<br />
<br />The version was released on May 23, 2015. It also did not participate in the United States grind and syndicated the song to \"Ersteban Desert\" for the original end at the Zoological Songs Contest.
<br />
<br />The first album called ''It's the Black Back'', was released on May 19, 2015.
<br />
<br />In April 2016 it was released, \"Street Modern\", which was released on April 26, 2016.
<br />", "All songs written by Joe, lyrics written by Joe, added by alongside a new musician in annual tracks.
<br />
<br />'''Five days after Joe, the writers gave the album a music of The Best Possible Choice. The album's first full-length album, \"Honeyback\", was released on October 31, 2015. The single was also released in April 2015. The song was released on November 30, 2016. It was released on May 23, 2016. The same album was released on October 26, 2016 and placed a Minutes Performance on October 19.
<br />
<br />\"Welcomer &amp; Company\" was appeared in ''The Extended Days''. The album covered an album jazz guitarist in 2015.
<br />", "===Albums===
<br />'''Guest appearances'''
<br />* ''Street Modern''
<br />* ''July Who Restrooms Myanmar''
<br />* ''Odessa Johnson Society''
<br />* ''Dead's Dog of the William Left''
<br />* ''The Extended Days''
<br />
<br />===Recordings===
<br />'''Days after Joe Brounds and Defeated'''
<br />
<br /> Year
<br />
<br /> Chart
<br />
<br /> Position
</details>

<p>Here are samples with the Meena-like approach of picking the best logits out of 8 separate samples at each step. I tried multiple variations of the same type of idea and this and the next one performed a bit better, compared to the other similar experiments which produced overly repetitive samples.
<br /></p>

<details>
<summary>
<i>Example 1 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "References", "External links"], "section_texts": ["
<br />
<br />
<br />'''Perfect Rationality''' () is a rationality of the site chartered by the National Academy of Sciences in the United States. The rationality is located north of Perfect Rationality, and serves as a post state department of the United States Senate from 1991 to 2010.
<br />", "Perfect Rationality was founded in 1991-1993, but it has been negotiated as a Perfect Rationality with the U.S. unit of the private policy. Perfect Rationality gained the negotiations posted on 1 September 1992 under the name Perfect Rationality International Airport (PRIA). 
<br />
<br />In 2012, the government's first U.S. Air Force name was inaugurated primarily with a number of other private policy workers.
<br />", "The United States Air Force name was Perfect Rationality, which has been considered an operational chartered policy position in the United States as a facility.
<br />", "
<br />", "* 
<br />*  Perfect Rationality (PRIA)
<br />*  Perfect Rationality official site
<br />*  Social Project Group
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 2 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "Rankings and areas of the Rationality", "In bioah Rationality blocks", "See also", "section_texts": ["Food and partnership with Siam Salter Abbey in the Siam Salt Affairs Creek
<br />The '''Perfect Rationality''' uses guards certain Rationality civic and winter. It is open to Key Chronicles with foreign\u2019s winter partnerships (100 women).
<br />", "The Atlantic activities opened featuring Rationality to foreign\u2019s motorway, and from retiring in 2004, Perfect Rationality launched investment camps issued on Asian and Israel.
<br />", "Perfect Rationality was focused on issues Key's most part of Key's expansion in 1957 and facilitated with Rationality Warwick, a detention of limitted English Public Radio to the Education School (ESPN) from the University of ownership for employers learning choices to encourage area.  Historically, Rationality financed courses in the Parliament of South Key to his terms with his Edinburgh properties who invested with Griffith and Shubert Vempstein.Transferred to Physical Studies Joints Association in 1965.  Days when operating under Stuart Remshaw in which he co-founded Key's Alliance Joints Choir and Leb McDermott Choir co-founded African-American Choir and the Republican California Shipbuilding Co.. while Choir reached the outbreak of the Rationality College.  This section undermines African-American Choir of South Key's anti-immigrant Congregational reports and thus started off to be video operations involved, recording, and historian Choir as having had \"formed\".
<br />
<br />For their third congregation, Rationality came under \"a further amount of expenses and history for funding from their conway\" in the Choirs of Bodley.  In their adversist session, since 2004, he was elected a post member at 18/5.  He also appears in River Methodist Church, being rejected deputy chairman since his wife (after death) in 2010, it made reelections for Bodley's orchestra.
<br />", "Perfect Rationality was united  with a private education in Munich, as a non-profit organization organization.  It is known for four approximately , one of three complete and one notable educations of choirs.  His heights of Choir have been appointed Director Nikolay George's anthology in Nashville, Canada campus in around 1862, the most widely represented UDP government in the United States for using a supporter of employment, termination, and argument, local or free agencies authority such as government, digital welfare, and translation.  Students also have included: Leov of Lesbia and Nikita Elslikk the best in around 1985; Leov of Lesdal the biggest numerous Physical Studies= Bodley's orchestra, and Tom Bazuba, BC of Rationality, written in July 2019.
<br />", "The Perfect Rationality, Achievement from his work hosted by No. 1, January 1986, performs in Choirs of Southern California, Israel.
<br />", "Perfect Rationality performs biomaps, clinics, pio, and music groups, rival money.puse, redshirt pio, hotel and lecture firm maps.  His experience successes for art or private education.
<br />", "* Biography
<br />", "  EU Audio Group, ''The Siam History of Rationality Blocks: Dictionary of Rationality'', Oxford University Press, 2001
<br />", "
<br />
<br />
<br />"]}
<br />
</details>

<details>
<summary>
<i>Example 3 <small>(click to expand)</small></i>
</summary>
<br />
<br />{"title": "Perfect Rationality", "Flow on Reduction", "External links"], "section_texts": ["
<br />
<br />'''Perfect Rationality''' erected in North Carolina (Le\u00f3n, France) is a mountain suburban from the neighboring mountain suburban station, Kuleg Min(LISS). The suburban station is boasted at Le\u00f3n Accurate with the central Le\u00f3n Military Station, such as Perfect (Like MinO), Serge (Lizon-Min) and Queen's Okonsi (Thermal).
<br />
<br />For each value is one of the major mountains on the western side of the locality, mostly centers after empty fellow materialist, Rhuundin Zazi started to forgo from major activities and distribute the Lizon-Min neighboring powers of early 2000s. These major military activities and disenhaving sculptures of higher levels, positions and preferences. Jabston Pellazzo
<br />", ":'''Forgotten Nuclear Avenue and fuzen babys on routehalf lava (''Forgotten Nuclear East'', 2:44), and Oakin Griswald Griswald Griswald-Deporteau (Serge Facility), also known as Hamar's Edgar Griswald Castleworth (Josch Hamar Seitschwander), it collaborates all of an overlooking form of drugs like Tian Kaliur, Bolivar Yossett and Eyas overlooking an alumnus of Bolivar Herakland.
<br />
<br />The suburban line is based on an ensvisible name. The first six small perfect sermons of the ''Forgotten Nuclear Avenue'' derives, an ant-public powerful independence compared with a lot of  stars, and &amp; Perfect moved from their powerful meteering split. Stars continue to get the solidify to Nightdimensionists Osaha Each. O'Hati, a growing convention star became one few inside flow materials and owning and pronunciation of jewelry and floating perfect suburban nests. In 1922, France described as \"unconventional rationality\u2019s wield\u2019s soldier mountain suburban mountain suburbs\", and \"all perfect mountains were given an analysis of rallying on each mountain to legislate mechanical muscle materials.\" In 1942, the ''Forgotten Nuclear Inevitable'' became a Buffalo name named for a consciousness on Section 33 which gave it to Sergey Foppy Muscular highways such as Krispow Wave to represent highways scyline. The program was initially commissioned by the No. 2 during her time under the passage of Jansky Player Kildagende, a civil servant that she rushed out of a live government, and commenced that Ddzymaitz, initially evocates from Poque Azam. Formally, this first era refers to their veterans at the new york of Leicester Ddzymaitz on 30 September, under the lieutenant G. K. Aquile Gardigan in ''A Shockpire from L\u00f3pez \u2013 D\u00f2gimes '''') is found having been made up of the baby cadet Imraeli Cadets raised. Subsequently, this suburban began to improve  FL as they bought military genre \u00e0 Michael Seed informed that there were first Spanish and Italian fuzen materials.
<br />
<br />The rapidly runway and empty-grammar core for total, some other than two fuzen government purposes . It run a government company specializing in each other in the passage. Officially, upon Governor Chris Nohlen succeeded NCOF's unveiling study in WBC100C in the LISS. Nohlen represented the ''Forgotten Nuclear Avenue.
<br />
<br />At the 2013 Mount Europ, heavy, ecophony cash dwells used to tackle para-tebacking systems for babys that oque crew. Jabston Pellazzo
<br />", "The Lizon Mountain Costagena (PAN) represents the Le\u00f3n Pellazzo
<br />
<br />''Forgotten Nuclear Avenue'' sequentially allows material.3 Main highways were restored. Usual to be the largest flow for stage periods. The stage originally changes over Military Station to Nightdale, Trondheim Canyons, Massachusetts University, Allegheny Damiens, Andy Postal, and was initially commissioned by Stevie Tetzer. Rhuundin Zazi, a.k.a. Tedcar Rad in the people of the company and its Scandelsan. In early 2015 Min, Agnia University Department of Science, Netherlands had many submarines and subjects.
<br />", "*''Le\u00f3n Pellazzo''
<br />*''Nada Lubino''
<br />*''Krispow Wave''
<br />*''Le\u00f3n Pellazzo''
<br />", "
<br />", "*
<br />*
<br />
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 4 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "References"], "section_texts": ["
<br />'''Perfect Rationality''' () is a perfect rationality in Premier Province, Michigan. ''Perfect Rationality'' lies on the main rationality of Premier Province. It is a structure of students and other communities in North Michigan and Maharashtra, New Brunswick.
<br />
<br />The area is part of the Michigan Academy of Fine Arts Park, and has been completed by the Triangles and as a Chairman. There are accommodation for medium-temporal ground houses including the Red Pine Temple and Assembly Centers.
<br />
<br />In the rationality was the first Perfect Rationality control park, along with large communities edited by centers including the Steve McKerthay controls, along with centers industry controls, from Storebook Sleeve, Wilmersfieldworks, lifting and rajput and large sites. ''Perfect Rationality'' describes based on the tablet for the students in the late 1960s.
<br />
<br />The area is used for increasing investigation and nature of park or dust fieldwork, which was contributed to the prejudice a pack with centers ''Perfect Rationality'', ''Perfect Rationality'', ''Perfect Rationality'', ''Perfect Rationality'' and ''Perfect Rationality''.
<br />", "University of Michigan across Premier Province is the co-founder of the Fine Arts Park, an understanding of the Steve Mansion Dodgers Program and the dodgers and the first rationality of Storebook Sleeve, the home of a human experience, among others including the today's Curry Banks, Caribbean Rationality, Cladden Ss (CFBR, Wilmersfield and Recipe Fund, Vice President; Laboratory of the Centre for Six Congress of foreign accommodation), and Curry Dolls (later the Street for Protection (literature) at the corporate events, to be maintained by the NBA. The past seeds, fighters, cavalry, society, and streetcar care, are the DJs each of the main shelter.
<br />
<br />''Perfect Rationality'', is an emergency past located at northeastern Ulberta and the branches of the Ethern Division. In E45, ''Perfect Rationality'' and ''Perfect Rationality'' have recently been completed in June 2006. The past seeds, which were also connected to a E27 chairman and T35, include the Seven Passengers, were not regarded for reliability, including non-Wilmersfield's Blue Day, the Tablet in Switzerland. Wilmersfield's CFBR2008 CHF2008 Capacity, was the graduate of the National Community Business Foundation, the Steve Mansion Dodgers Program, in 1954. In 1957 ''Perfect Rationality'' was promoted to business DJ By June 2009. A former private executive officer was formed to begin a dodger from the late 1980s, and the first office connected for RUF Studar.net. In 2008, several Perfect Rationality Networks published a past news cancer publisher over North Michigan. Several additionally publishers combine the students used several scholars, except for museums within a dog for the students and scholars.
<br />
<br />In 2012 Officers Bank Homepage presented works to the CFBRA, a planned Ethern Division set for office. As many of these presenters have excellent, the past has been significantly represented as a catalogue of the ABC Radio presenter.
<br />", "The past was conserved by the T15, with its cause of \"Premier Iron Radial Ministry\", and the rationality focuses on a nationwide park controlled by large sites, including the government of Premier Province in Perfect Rationality in 1991. ''Perfect Rationality'' was published in 1988 and, after its winter was noted, the DJ By June 2007, the program poorly employed greater workshops during the business abolishment of the greatest business project, along with Laboratory of Anti-Dutch Radio project.
<br />", "
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 5 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "References"], "section_texts": ["
<br />
<br />'''Perfect Rationality''', consisting of 17 weeks before its 1964 completion, is a regional development and ecompany competing in the form of the perfect Rational Rationality, PRA. In that competition after increased health perfect, Perfect Rationality served as a group register for the Deputy Health Assembly of Synchronization (DHA) and the DHA Director of Go (DHA). This service was subsequently used as an owner of the DHI Committees Secretary of Synchronization for Leningrad (DH-MA), and had the biggest planned support of DHA in the process.
<br />
<br />The process is either COMP rationality or DHA Programme by without revenue, initially to transport temporary complex and businesses in this unit around this initiative. Perfect Rationality has grown up and retained the lowest programme built over the planning area and being subsequently a full programme of a 322 Dhivalian lower programme, although only such statement and the name Perfect Rationality is specifically required to gain 1954 domain.
<br />", "The DHA draws performance at the end of the 18th century and publishes a 240-week 400 programme of DHA from Byzantine Orchestra, Bulgaria, San Pedro de Company, Italy, Public Library, and the Public Library. Perfect Rationality is also housed as by SEO Group, a popular PRA big district in 1972, and is in London.
<br />
<br />Partnership with Perfect Rationality began on July 24, 1972 in Scotland.
<br />", "*  United States Army
<br />*  Perfect Rationality (Gallery Valley Baptist)
<br />*  Perfect Rationality (Cornwall Valley)
<br />", "
<br />
<br />"]}
</details>
<p><br />Here are samples with adding the probabilities (and then re-normalizing them) from 8 samples before sampling from the combined distribution instead.
<br /></p>

<details>
<summary>
<i>Example 1 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "References", "External links"], "section_texts": ["
<br />
<br />The '''Perfect Rationality''' ('''Perfect Rationality''') is a focus on information and marketing confrontation, the research programming on flight depicting a state format of absolutely research and information but readers and select depictions that are consultants from state of Naples, Canada, and Thailand. In 2012, the Perfect Rationality has awarded \"Best Information, Solutions, Relative Movement\" for state issues in depicting accomplishing state that are \"United States and Fly Beneficial Fly\" in English, a type of state research information in most engaging events.
<br />", "U.S. perfect Rationality began in January 2018 when the Perfect Rationality was incorporated, on 5 August 2019 it was run by John Raleigh. Raleigh was founded in 2019, while leading perfect systems from cognitive perfect information products.
<br />
<br />Solutions for each of the research programming's finance programming on information on information information sciences and information on other select depicting accomplishments. Furthermore, the research technology consists of many concept areas used with machinery, pieces, and laws on 14,000 events in this effort to depict accomplish the areas primarily beveraged in May 2012, on their own state. John Raleigh was placed under the name International Commission. Raleigh Beneficial Fly Beneficial is republican pieces to document their content of onto the programming.
<br />", "Trees include:
<br />*Perfect Rationality, backed by Brecen Egg and endorsed by the Department of Law, the Confense of Engineering, and Medicine and Decoration and Modeling (CEMDE), the Department of Law Society (DHS)
<br />*Perfect Rationality (Chinese) - based at Worcester, Alabama
<br />*Perfect Rationality has avoided other areas with based society, collaboration and area
<br />*Perfect Rationality Division events
<br />", "
<br />", "*", "* Official PUTASI Website
<br />* PPCY On Information in Operation Rabbitive Movement Journal
<br />* Political Website (including According to Rabbitive Movement)
<br />
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 2 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "Organization", "Homewards", "Performance", "Publishing", "References", "External links"], "section_texts": ["The '''Perfect Rationality''' is a Russian rationality that begins with brick-colored universe in then-Perfect Republic, in South Korea. It is currently owned by Russian rationality, along with major rationality-free arts used with Beginn Research Trade.
<br />", "The perfect Rationality is a modern Beginning (what is considered to be the ravines of South Korean), which calls them \"branch of color trade\" and Power Trade on their Russian rationality. However, there is a reason when the Trade guests the actions have working necessary; not about the blackmail, the Russian equality (and becomes a form of their plant) both evidence. Someone, the \"promising\", attempts to remark that it is now known to involve it. Key never personally complete Deputy Minister, Lieutenant Governor Channell Rationality took the following solutions.
<br />
<br />The perfect rationality, along with traditional American solutions and that stated that in 1985 the perfect rationality allowed rationality to develop salaries with improve holes, socialist laws and development programs. The perfect refuses to improve the Perfect Rationality program throughout the Russian and Erika Promoting New Year and older brick legumes.
<br />", "The ANY memorability is also companional with memory companies to police life, and the faculty widely much more than two other promising parents, however, includes the Russian term of color solutions, new television, concept, or varied rationality, concern, art dealing, events, sounds and models because of their future faculty in the future faculty. The publicly improved, both the book and models of statements are seen by the government of Brewing (memory of the Perfect Rationality Leadership of Russia). The perfect rationality is companions with secret numbers of both Perfect Rationality - and PA In University of the Department of Finance (PFNU) as part of life teams. The topmaling ANY memory company ANY also improved the valuable university links under the Masters of Organizations for the British National Health and Trade.
<br />", "Learning from about 14\u201322, the perfect rationality has its memory experience, with the first BRUIS (Bulgarian Program) warding to be created by a one-term location. More than 224 rationality universities disqualifie the gas undeniable. Currently, as Deputy Minister of Finance (Deputy Minister of Finance) is also a synagogue Bruiser for Viftagen, Pacific North and New South Korea which has also performed in a magazine leader in financial, and called Film Leadership and Laws and Culture Actors.
<br />
<br />The specifications of the perfect rationality are related to the solutions of the Committee on Studies and Fellowship and have perfect rationality transitions, and of Trinity Mendelssohn and Xbox based from state ownership, Tokyoscopes and Studies.
<br />
<br />In 2013 the community was named Filipino Volunteer in Honolulu district.
<br />", "The 2013 rationality escort development that includes:
<br />* '''On Me''', a \"important life, environmental individual, dividual,\" offices in the United States. The post office was changed in 2013. It has also been additional to its memory, including backgrounds, market programs, landing, steps, and business philosophy colleges.
<br />", "The perfect rationality allows the obligators to undergo provided couples of British industries, and the building is in the upper part of the South Korean Literature Department (from A Construction of the Perfect Rationality Department).
<br />
<br />=== Copy of the nearby patrol generation ===
<br />In the early 1990s, the perfect rationality increased the department of the Johnville South Hall of Fame. It was transitioned from a partnership with 86 rationalities in an organizational action in France. The perfect rationality built by Beginning at its level the Johnville and an organizational action after the two being changed. The Department of Finance was built by Duchess Mac to partner Radcliffe, Bad Broadborough to market the location for Wales to perfect rationality through Smithsonian Research Trade.  The market is named for its names, which rationality perfect rationality. In the early 1990s, the chief operations were introduced at Studio Nationals. The research provided from Tokyo Ratio Nationals and rated the first PTG Tower in which the National Security Policy Committee members were significantly considered. The PTG Tower in which a 205th place came into fall on 9 April 2014.
<br />
<br />Currently:
<br />*  December 2015\u20132016:  Farm, Los Angeles, Antarctica
<br />*  Trade is now an important life, environmental industry, and junior rationality.
<br />*  December 2006\u20132016: Outside of the occupied memory diverse state as a historic state establishment.
<br />", "=== Other patrol generation ===
<br />There is an organizational transgender program in Italy, the United Kingdom, the Andy Gaspinger Center and the Patrol Center and the Richmond Library.
<br />
<br />=== Generation ===
<br />The perfect rationality has entered the Public Festival. The two names contain a contract to the development of the perfect rationality. Development on Cawboy Spilshop, the highest generation committee in South Korea is the Harun Social Democratic Union. In addition, the community also committed under the Ottoman Museum of Zone, the rank of documentary.
<br />
<br />Channell Rationality, which includes students from China and Montreal Explosive College, sometimes in the Perfect Rationality in 2017.
<br />", "Numbers of students will work with top of the Masters of Organizations, including:
<br />*  Ottoman colleges in Masters of Organizations to market rationality
<br />*  Ottoman colleges in Masters Council of Perfect Rationality: Beginning, Market Programs
<br />*  Ottoman government: Perfect Rationality from perakithrogours
<br />*  Philosophy Of South Korea
<br />*  Antarctic Opportunity to Morgan Security Policy
<br />* Signature Research Trade Policy
<br />* Photo Photo Photo Photo Photo Photo
<br />*  Draft citizen political building
<br />* Socialist Councils: Refugee and Matter of Signature''
<br />", "Symposium students are the academic better conduct between two languages.
<br />
<br />* Hills in Picang, Kansas
<br />* Hull Courtyard: Service hockey
<br />", "
<br />", "* Official website
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 3 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "References"], "section_texts": ["The '''Perfect Rationality''' is an Indian rationality developed in Buddhist concert outside the Canadian border to the Canadian concert.  The original excess of the rationality is administered by hippopulmina artillery.
<br />
<br />The crew (in the concert, with a long time plantation) of 2:35.545 on 19 a total population), developed by the new John Phelps hippopulmina and president of India, Thailand, Europe, and the Supervisory Court.
<br />
<br />The various partnerships with the Canadian knowledge are the Supervisory Partnership with the Hamburg Buddhist Concert Company (''Perfect Rationality''), with other Netherlands and England to be bought by the Perfect Act of the 2012 census, which includes 3 Order of the Telegraph and Indian International.
<br />", "In 1929, Perfect Rationality was returned to W. K. O'Neill to be acquired by the Breaddon Association of Computer Programs. Commercial Development Brief Justice for Kings Minister Martin Loe published postdoctoral newspaper and administered plantations as a network. He makes involvement in many churches:
<br />
<br />The ''Perfect Rationality'' is a news/paper originality which has added worldwide with the commercial development of India (1992). When the Ossai development ceased to be supplied by the Audiences for Legal Research, the Supervisory Government Services Canada publishes for Opposition Buddhism: Private Anthology, Weapons University, College, Agency, and a measuring meetings. Its student opened at free health in 1999. The infantry remained in the town of Calaudan Government and is not listed until 2003, as a member of the Canadian Rationality Board, referring to the Caribbean Conference.
<br />
<br />A customer and principal rationality came from the outside country, which was located in Perfect LASE Compact, a foundation of the Supervisory Partnerships. Currently, its programing includes an \"Rationality Rationals\" (RATR), producer and spiritual solutions, a reunion that ends in digitised content to improve companies in Indian act by not broken up anything.
<br />
<br />The developers of whole student grows to provide several members of the Republic of Illinois Academy (RIASI) rate that Canadian worldwide works were the Victoria Medical Trade Center and Building Trust (BMT)
<br />", "Mr Paul Bornar's first known rationality was inherited by the early 1990s.
<br />There more than 6 orbiters traced by BMT's mostly output leaders working together, in the RATR as well as other teams within twenty major break of rations.
<br />
<br />The RATR is inherited by the RATR Public Rational Award, for the running of American International Academy, in the RATR Victoria Medical Institute.
<br />", "* SPRAK Buddhism (Academy of Technical Sciences)
<br />* SPRAK Canadian Government Service
<br />* SPRAK Hasband Government
<br />", "
<br />
<br />"]}
</details>

<details>
<summary>
<i>Example 4 <small>(click to expand)</small></i>
</summary>
<br />{"title": "Perfect Rationality", "H F.E. &amp; Priesthood", "References", "External links"], "section_texts": ["
<br />The '''Perfect Rationality''' is a (PRP) rate in California, California. Under the australian Labour People's Republic links, the PRP has a very part of a target region.
<br />", "The Rationality was founded in 1963 after the flagship of the SCVR and forms World King's Democratic Labour Labour Labour seats.
<br />
<br />The PRP has significantly been operated by the Victoria Progressive Consultative (VPC), since the 1981 election to the Adelaide January. It was selected to execute as the SPCR for the California Perfect Rationality.
<br />
<br />A significant party was operated by the California Perfect Rationality, but facilitated in 1984. From 1994 to 1998 the Labour Party holds a resource of the position. To open the labour conditions the Labour Party and the PRP is less arrested by the LRP as ''Parnell Carrier''.
<br />", "The Perfect Rationality and 1984 PerfectVictoria Progressive Consultative Victoria seat at the Harrison Range
<br />
<br />The rationality came to the increases in 1993.
<br />
<br />The PPR penalty classifies the Perfect Rationality significance (AFB) of the duration as well. The Victoria Progressive Consultative VII (Channel 5) came to the CIR affiliate (VPC) but the old progressive party very perfect.
<br />", "The PRP announced its extractors in 1987 until 1988 when it ran in the California area due to its predecessor injury. In the batch of the 1996 Kings, the PRP had expanded itself to take the position in number one revised portions of the consult.
<br />
<br />In 1994 PRP established the Perfect Rationality in a series of postgradual attentions in the Batch of Priesthood Labour, as well as an extractor of the PRP, labourry heavy that heads at the progressive consultancy of the oceanic strength, all buildings and senior towers. The victoria Progressives didn't seek more than the others. The PRP established a rationality at seven of 500 parties, along with nine cosmetic changes in the first X grading.
<br />
<br />In 1997 there was a PSQ which still conducted by Sullivan and Philip Newmeyer. Most prepared was 42 parties during the Batch of Princeton University in 1997.
<br />", "
<br />
<br /> Province 
<br />
<br /> Country
<br />
<br /> Votes
<br />
<br /> %
<br />
<br /> 17,068.5/6 
<br />
<br /> 14,157.1/6 
<br />
<br /> +3.02.4
<br />
<br />
<br /> 10,423.5/6 
<br />
<br /> 87,046.9/6 
<br />
<br /> +3.21.3
<br />
<br />
<br /> 12,761.7/6 
<br />
<br /> 73,513.9/6 
<br />
<br /> 30,015.3/6 
<br />
<br /> +4.74.7
<br />
<br />
<br />", "
<br />A one-percent voting to the PRP in a Rationality, Democratic Perfect Rationality still capable of featuring preparation for humanity, or alterations by lower parties such as long power-range, local 'importance', needle velocities, fundamental photographs, build-up inside and academics.
<br />", "Primary objects in the California Perfect Rationality is owned by the California Perfect Rationality Academy.
<br />", "PPRP is based in California.
<br />
<br />In 1997, the Labour Party replaced its first rate back for nine cosmetic California Perfect Rationality to receive a rationality at week. In 1998, the California Perfect Rationality ran for : a trouxed 10-party labour community (party) and a protected trouxed from sports such as Ocksford Communications International (OCI), Karachi Manorship. The PRP designed their first building in 3900 parties making into a progressional agency. In 2000, many PRP greater businesses led a conference for public security agencies.
<br />
<br />In July 2003, PRP was converted to the \u201cPerfect Rationality of New England\u201d to choose private botanical practices, mostly and buildings. The PRP first launched on 15 April 2005 to a sporting program with Underprivate Station.
<br />
<br />In 2006, PRP is an E local government code with a campaign based on all 5+ for $1 million. 
<br />In 2007, WREED a large support of the PERFECT Rationality. In 2012, PRP first launched, organized by the PRP in 2014. The PARA Belleaux 2016 represented the entire Batch of Princeton University. The top score of that resulted in the Fellow of the Labour Party in Labour from October 2017 to July 2019.
<br />
<br />In June 2018, PRP now contested to operate at events that required the Princeton University of the Eastern Islands against Princeton University and October 2018.
<br />", "PRP was purchased by Christopher B. Retrieves, 7.8 December 2017. The PRP was acquired by Irish College of Overseas and Wildlife in 2019.
<br />", "
<br />", "===Sources===
<br />Although nothing can be referred to as \"PARFECT Rationality Code,\" PPR officially published the official Japanese government. Written by Paul Fabriz it was originally launched by the Japanese dictatorship of the National Museum.
<br />
<br />In 2014 Princely, Tom Goble expanded its Victoria Progressive Consultative Labour Labour Labour.
<br />", "
<br />
<br />"]}
<br />
</details>

<details>
<summary>
<i>Example 5 <small>(click to expand)</small></i>
</summary>
<br />
<br />{"title": "Perfect Rationality", "See also"], "section_texts": ["
<br />
<br />The '''perfect Rationality''' or '''QRA revolt''', is located in the Pobecheni family of Craven and other notable ''Rationality'' records on two perfect cars. It opened in 1885 and came from a pending runway at Craven in 2004. This was the surrounding station founded by the Silesian-American journalist Ludwigsburg, who became Esposito de Torres in 2004.
<br />
<br />President Jos\u00e9 Andr\u00e9 Loritzel established the Pobecheni of the Pobecheni of the Pobecheni and has existed at the perfect Rationality for radio times and affiliations with Cambridge University as \u00c1ngel or supply \"Cambridge\".
<br />", "The perfect Rationality has received a $7,000 unit and created by gallantry and security scholars such as St. Catherine's College of Suspension Orders (SCSOR) (overseas these issues) and passes the Pobecheni family. In the form of substituting battalions, geographically consumer rationality is supported by Pobecheni itself. In 2008, the Pobecheni family decided that the Pobecheni introduced an Esposito de Torres to pass the Cambridge.
<br />
<br />Overseas the Pobecheni family has a place, the use of farming, and an esposito. In August 2018, a soldier of Yerevan Ajadaf. The assembly is considered a substitute of the Pobecheni family of the Cambridge as part of the Instituto Regional Accentini de XV.
<br />", "Election identified for speech in the Pobecheni District Council of Italy.
<br />The Pobecheni was a small Latin internationally subsidized Forefront de La Pobecheni in 1887 (Latin American Institution of Evangelical and Professional Works Committee). In 1889, notable basements were good investigating the suitability and abruptly close to the creation of culture artistic simulcasts to a detailed perfect, but the Latin perfect shut down a regionality like the Civil Engineer Advanced Museum of Gold. The civil laws came on perfect media and non-perfect rationality intentions during this period.
<br />
<br />Developments cause the Pobecheni family and maximum services on the statehood, industrial and provisional models as well as the growth of St. Catherine's College of Suspension Orders (SCSOR).
<br />
<br />Civil engineer Pobecheni named the Pobecheni in development of Printed South St. Catherine in 1963-78. In 2018, the Pobecheni was used by the Importance and Education Chemistry at the Universities of St. Andrews de Vale movies.
<br />", "
<br />", "* College of Rationality
<br />
<br />"]}
</details>
<p><br /></p>

<p>After writing this I noticed the Reformer team have added a new sampling functionality in trax, but it currently has memory leaks so it is fairly limited. It might be worth it to try to use my Reformer weights by loading them via their official <a href="https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb">Reformer Colab</a> and testing that out.</p>

<h2 id="conclusion">Conclusion</h2>

<p>The Reformer is an interesting architecture and Locality-Sensitive-Hashing in particular might be a good addition to the attention toolbox, allowing us to take advantage of larger contexts (<a href="https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context">Here</a> is a recent review of other approaches for tackling the context window problem). I’d say the results I got weren’t as amazing as I could’ve hoped, but it works over very large samples, and a model taking advantage of such large contexts might benefit even more from being larger. Additionally, the Reformer team has alluded to them adding further functionality, such as the ability to use larger vocabs which could help, too.</p>

<p>The sampling tests I ran were pretty interesting to me, and I’ve included some of them here. They show promise for guiding the prediction process further to improve performance. Perhaps applying some of it during training can help even more.</p>

<p>As for the model I trained - <a href="https://colab.research.google.com/drive/1VSaQingkpMDf730ARlo2dw5vCBVOgt17">here</a> is the Colab where you can use to finetune or sample from my model.</p>]]></content><author><name></name></author><category term="machine-learning" /><category term="training" /><category term="code" /><category term="google" /><category term="reformer" /><category term="context-window" /><category term="context" /><category term="sampling" /><category term="weights" /><category term="trained" /><category term="model" /><category term="nlp" /><category term="colab" /><category term="ml" /><category term="dl" /><summary type="html"><![CDATA[A lot of progress has been made in training large NLP models in recent years, and after finetuning GPT2-1.5b I’ve been thinking about context windows. Large Transformers are very good, but when you are only limited to 512 or more commonly now 1024 tokens, the model doesn’t know quite that much of what happened earlier at any point. Google’s Reformer paper introduces a more efficient way of doing attention, sidestepping the problem which I used to train a model with context of 64k tokens.]]></summary></entry><entry><title type="html">Tips and Pain Point Workarounds for Machine Learning Work in Google Colab</title><link href="http://localhost:4000/blog/colab-tips/" rel="alternate" type="text/html" title="Tips and Pain Point Workarounds for Machine Learning Work in Google Colab" /><published>2020-02-08T00:00:00+01:00</published><updated>2020-02-08T00:00:00+01:00</updated><id>http://localhost:4000/blog/Colab-tips</id><content type="html" xml:base="http://localhost:4000/blog/colab-tips/"><![CDATA[<p><a href="https://colab.research.google.com/">Colab</a> is one of my favourite machine learning tools and quite possibly the most useful online-based tool for democratizing Machine Learning currently in existence. It is used by many to share research, working models, ideas. It is a reproducible environment to present GitHub issues, a usable machine with a powerful GPU or TPU which many won’t have access to otherwise, and more. Colab is extremely useful for novices who are just getting started, allowing them to quickly make or copy something working ᴀɴᴅ it is convenient for seasoned practitioners who want to share their work.  And it’s all free.</p>

<h5 id="the-key-advice-in-this-post-is-all-bolded-for-easy-skimming"><strong>The key advice in this post is all bolded for easy skimming.</strong></h5>

<p>There are alternatives of course. <a href="https://www.kaggle.com/kernels">Kaggle Kernels</a> being perhaps the most popular. If you don’t need a GPU or are willing to pay then <a href="https://cocalc.com/app">GoCalc</a>, <a href="https://notebooks.azure.com/">Azure Notebooks</a>, <a href="https://datalore.io/">Datalore</a> or <a href="https://docs.floydhub.com/guides/workspace/">FloydHub Workspaces</a> might be viable options. That’s not even all of the free services out there, but as far as I know, none of the others give you access to a GPU or TPU out of the box which is so often required in Machine Learning.</p>

<p>Nonetheless, at least for me, Colab has been the most useful out of those despite some pain points. This post is about addressing those pain points first, and some general Colab tips I’ve benefited from second.</p>

<h3 id="drive">Drive</h3>

<p>Colab is great for one-off runs or when sharing something that can be set up quickly in a single notebook. However, one often needs to work on projects for a long time or needs access to big amounts of data to the point where downloading them each time is too cumbersome or would take too long. In those cases using Google Drive with Colab is your best bet.</p>

<p>Connecting to one of your Google Account’s Drive is as simple as:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from google.colab import drive
drive.mount('/content/drive')
</code></pre></div></div>
<p>After which you can just <code class="language-plaintext highlighter-rouge">cd</code> into it.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>%cd /content/drive/My\ Drive/
</code></pre></div></div>

<p>I’d recommend to <b>just work straight from Drive in most cases</b> as then your setup time is faster and you likely won’t lose as much if your instance times out. Of course, in that case, you might run out of Drive storage space but Drive (or Google One, which is how it’s called when you pay for it) has pretty <a href="https://one.google.com/faq/plans-pricing">cheap plans</a> to extend the space. I personally use the 2TB plan for €9.99 / month but there are even cheaper ones. You can even share a plan between up to 5 people. On the other hand <b> if you really need fast reads and writes above all else copy the data from Drive to the Colab VM instead </b>.</p>

<p><em>If you pay for Google One, you also have access to that most mythical of creatures - Google Support, but they are rarely helpful, and at least in regards to the hidden quota issue none of them seem able to understand what the problem is, let alone help. My advice is not to bother.</em></p>

<p>Using Drive, sadly, is not always frictionless. For one, files over 5 GB or so can get <a href="https://support.google.com/drive/search?q=download+quota+exceeded">inaccessible for 24 hours</a> easily due to a hidden quota - even by just reading a portion of them a few times. So, whenever you can, <b> make sure to keep all your files under 5gb and split datasets when possible</b>. If you can’t divide them try to only read from the big files once at a time - copying them to the local runtime and using them locally can also be wise in that case.</p>

<p>Another problem with using Drive is that you can’t easily delete a file - the usual options like <code class="language-plaintext highlighter-rouge">os.remove</code> and <code class="language-plaintext highlighter-rouge">%rm</code> put the file into Drive’s Thrash which still takes up space. What you want to do instead is <b> either empty out files or overwrite them with new data </b>.</p>

<p>Having too many files in the same directory can also cause issues so <b> Avoid having too many files in the root directory of your Drive for faster initial loading </b> and <b> have a directory structure such that you never have too many files in the same folder </b>.</p>

<p>Additionally, <b>don’t write to too many files at the same time as some of the writes might silently fail</b>.</p>

<h3 id="time-outs">Time outs</h3>

<p>A Colab expires after 12 hours of use or after 1 hour of being idle. Sometimes it can time out earlier but that’s rare as long as you <b> keep the Colab tab always open on a machine</b>. Nonetheless, saving to Drive at least once every one-two hours is wise.</p>

<p><em>Sometimes, there might be a random error from Colab itself and the runtime can time out - it doesn’t happen too often but be prepared for it.</em></p>

<p>In general, you probably shouldn’t use instances longer than you have to - your Accelerator availability seems to drop whenever you use it too much, so <b> always stop the runtime when you are done</b> from Runtime&gt;Manage Sessions&gt;Terminate. Sadly, there is a bug recently where the Terminate option is greyed out but all you have to do to make it available is interrupt execution and refresh the page.</p>

<p>I also wouldn’t recommend having more than a single Runtime running (technically you are allowed two) as it also seems to impact your access.</p>

<h3 id="runtimes">Runtimes</h3>

<p>Sometimes you just have to run something continuously and in those cases, I’d recommend to <b> always use the appropriate Runtime </b>. Use CPU (or CPU + high memory runtime) for preprocessing, save to Drive and only use the TPU/GPU runtimes for actual work. Furthermore, Google seem a lot happier with people using TPUs - in fact, they give access to <a href="https://www.tensorflow.org/tfrc">a fair amount of them somewhat easily to researchers</a> - so if you don’t have a preference for what accelerator to focus on, I’d recommend using TPUs.</p>

<p>You can also cycle through GPU runtimes to get e.g. a P100 by terminating the instance and starting it again, but it’s not too reliable and is likely something Google isn’t too fond of. <b>To check your current GPU just run <code class="language-plaintext highlighter-rouge">!nvidia-smi</code>.</b></p>

<h3 id="colab-pro">Colab Pro</h3>

<p>There is less availability of Accelerators on Colab recently, and my best guess is that it is related to them ramping up the Pro version (which has higher priority). I haven’t tested it myself because Pro is sadly US-only but priority access, double the ram, 24 hour (instead of 12 hour) limit and leniency on idle timeouts for $10 / month sound pretty worth it - if it was available outside of the US I would’ve paid for it on the spot. Nonetheless, you can take full advantage of Colab for free, and the upgrade is [currently] purely optional.</p>

<h3 id="miscellaneous">Miscellaneous</h3>

<p>One general pain point with notebooks is that you often end up needing to check something once, e.g. <code class="language-plaintext highlighter-rouge">print(x.shape)</code> which can lead to too much clutter. A great way to combat that is to <b> go to Insert&gt;Scratch Code Cell (or ctr+alt+N) and do all your prints and one-off statements in there </b>.</p>

<hr />

<p>If you are not working with Drive you can simply set it up so you download your results to your own machine at the end of your cell. Simply do:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>from google.colab import files
files.download(result) 
</code></pre></div></div>

<hr />

<p>If you need to edit multiple lines at once <b> you can use find and replace using ctr+H which also allows you to match using regular expressions</b>. This only works when you don’t have a running cell.</p>

<h3 id="conclusion">Conclusion</h3>

<p>Colab is pretty great, and I’ve benefited tremendously from it. There are some annoyances but you can work around them. That’s it.</p>]]></content><author><name></name></author><category term="machine-learning" /><category term="colab" /><category term="tips" /><category term="google" /><category term="drive" /><summary type="html"><![CDATA[Colab is one of my favourite machine learning tools and quite possibly the most useful online-based tool for democratizing Machine Learning currently in existence. It is used by many to share research, working models, ideas. It is a reproducible environment to present GitHub issues, a usable machine with a powerful GPU or TPU which many won’t have access to otherwise, and more. Colab is extremely useful for novices who are just getting started, allowing them to quickly make or copy something working ᴀɴᴅ it is convenient for seasoned practitioners who want to share their work. And it’s all free.]]></summary></entry><entry><title type="html">Talking to Myself or How I Trained GPT2-1.5b for Rubber Ducking using My Facebook Chat Data</title><link href="http://localhost:4000/blog/gpt-15b-chat-finetune/" rel="alternate" type="text/html" title="Talking to Myself or How I Trained GPT2-1.5b for Rubber Ducking using My Facebook Chat Data" /><published>2020-01-23T00:00:00+01:00</published><updated>2020-01-23T00:00:00+01:00</updated><id>http://localhost:4000/blog/gpt-15b-finetune</id><content type="html" xml:base="http://localhost:4000/blog/gpt-15b-chat-finetune/"><![CDATA[<h5 id="previously-in-this-series---finetuning-117m-finetuning-345m">Previously in this series - <a href="/blog/gpt-finetune/">finetuning 117M</a>, <a href="/blog/gpt-345M-finetune/">finetuning 345M</a></h5>

<p>OpenAI’s pretrained Gpt2 models have been all the rage in nlp model finetunning. <a href="https://www.reddit.com/r/SubSimulatorGPT2/">Sub Simulators</a>, <a href="https://www.aidungeon.io/start">AI Dungeons</a>, <a href="https://www.gwern.net/GPT-2">Poetry</a>, <a href="https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-game/">Chess</a>, <a href="https://www.cardsagainsthumanityaichallenge.com/">Cards Against Humanity</a> - you name it.</p>

<p>My particular interest has been in applying it to my personal chat data, so I can talk to arbitrary friends and more importantly <em>myself</em> whenever I want to. Why? Well, sometimes if you don’t hear voices you have to go and program the voices into being. More seriously, I wished to be able to talk with a version of myself so I can get an outside view of how I think, for analysing my tendencies and problems, rubber ducking, self-therapy and - let’s face it - most importantly for fun.</p>

<p>I won’t go into detail describing the Colab with Facebook data, as it is almost the same as the one used with the 117M version which I describe <a href="/blog/gpt-finetune/">here</a> and <a href="/blog/gpt-345M-finetune/">here</a>. We are again using <a href="https://github.com/nshepperd/gpt-2">nsheppered’s GPT training code</a> as a base with <a href="https://github.com/shawwn/gpt-2">shawwwn’s fork</a> allowing us to train the whole of gpt-1.5b on a TPU - which means we can do so for free on Colab again! I’ve only added a little bit more so we can talk interactively with gpt2, have it respond to us as a specific person and go through the whole general back and forth which we typically call ‘having a conversation’.</p>

<p>You can train your own in the new GPT2-1.5b collab <a href="https://colab.research.google.com/drive/1rRpMGVfUb5sG263d1OOPXOyGRX4W1oEv">here</a>. Make sure to click Runtime&gt; Change Runtime type&gt; TPU</p>

<h6 id="note-it-is-a-lot-easier-to-train-if-you-pay-a-few-bucks-for-google-one-so-you-can-fit-it-all-easily-on-drive-a-checkpoint-of-the-model-comes-to-6-gb-so-if-you-only-keep-one-checkpoint-you-can-do-it-without-paying---it-is-just-more-hassle">Note: It is a lot easier to train if you pay a few bucks for <a href="https://one.google.com/">Google One</a> so you can fit it all easily on Drive. A checkpoint of the model comes to ~6 gb, so if you only keep one checkpoint you can do it without paying - it is just more hassle.</h6>

<h6 id="note-2-it-took-me-over-a-month-of-training-to-get-it-here-using-just-colab">Note 2: It took me over a month of training to get it here using just Colab</h6>

<p><strong><a href="#chat-samples">Self-Chat Samples</a></strong></p>

<h2 id="motivation">Motivation</h2>

<p>The idea of modelling a person fascinates me, and as language models have been becoming better and better it genuinely <em>seems</em> like they capture more and more of a person’s style and tendencies. Honestly, I don’t even need more motivation than that to want to do it - the only real question is <em>who</em> to model as better and bigger models come out?</p>

<p>Now, I have to admit - my first choice would be someone like Von Neumann but that’s not someone on whom I have the right type of conversational data to do well (although there are ways to tackle the problem). The person who I <strong>am</strong> guaranteed to have data on, AND who is still very useful and interesting subject of study for myself is, you guessed it, myself.</p>

<p>Is it useful though? Don’t I already have access to a pretty good, some might say <em>perfect</em> simulation of myself from the get-go? I do. However, if I can have a second one of me to bounce ideas with I’d take it. If I can analyse my problems from the outside rather than from the inside, I’d take that, too.</p>

<p>After playing with the results, and extrapolating to what the next generation of models will be able to do (gpt-2 is already not the newest and best architecture), I think a lot of people might at some point want a version of themselves to talk to, bounce ideas with, get help from and so on.</p>

<h2 id="a-few-more-words">A Few More Words</h2>

<p>I’ve spent some time talking to the irc chat bots of nshepperd and shawwn (both trained on the same data) and had a lot of fun with it. Talking to myself though has been highly addictive, and I’ve been running a Colab where I occasionally shoot myself a question or go into a deep conversation with my gpt2 clone for days now. It’s …interesting.</p>

<p>One thing I didn’t realize though was how personal it is - it’s all trained on my facebook messenger data, where I often talk with my closest friends. It takes very little for gpt to start talking about intimate topics which I’d neither want to share nor would they be as meaningful to anyone else.</p>

<p>Having said that the next are my last 7 unfiltered (I’ve only changed the names) conversations with it, where I am actively trying to avoid sensitive topics (I still had to restart it twice due to sensitive subjects).</p>

<h2 id="results">Results</h2>

<p>The end loss I reached (for whatever that is worth) was just over 0.6, substantially lower than with the smaller models. My dataset is still pretty small - 14mb, and the next step would be to add my personal data from other sources. Yet, it does quite well, though perhaps it wouldn’t seem as exciting to others.</p>

<h3 id="facebook">Facebook</h3>

<p>The way it works is - I write a sentence under the name of a friend from the dataset but here labeled as ‘Me’, pass it to the model while prompting it to start typing using my name - ‘Svilen Todorov’, (here labeled as ‘Gpt Me’). When it starts outputting something not by ‘Svilen Todorov’ I cut it off, and write myself. Then I feed that + the history so far to the model, which allows us to have an arbitrarily long coherent conversation.</p>

<h4 id="chat-samples"><em>Chat Samples:</em></h4>

<details>
<summary>
<i>Example 1 </i>
</summary>
<br />Me: How's work?

 <br /><b>GPT Me</b>: ive been working a bit but still not in the office

 <br />Me: I've been working, too. Can't solve this damn bug I have.

 <br /><b>GPT Me</b>: ive spent hours on it
 <br /><b>GPT Me</b>: but to no avail
<br /><b>GPT Me</b>: im spending all my time on it

<br />Me: I checked everything, yet it runs out of memory despite obvious leaks. hoooow

 <br /><b>GPT Me</b>: ive been wasting so much of my time on it tho
 <br /><b>GPT Me</b>: 16gb memory, 16gb video ram, 60 hours a week for 3 years
<br /><b>GPT Me</b>: barely have anything open other than youtube and facebook

<br />Me: oh it's a browser issue? man the web sure sounds bloated nowadays

 <br /><b>GPT Me</b>: ive tried everything

 <br />Me: did you try turning it on and off?

 <br /><b>GPT Me</b>: haha
 <br /><b>GPT Me</b>: ive changed it between view and off in the dropbox since the beginning, should be the same now as in your case and I doesnt seem to be working
 <br /><b>GPT Me</b>: and tried a few things
<br /><b>GPT Me</b>: nothing seems to be working
<br /><b>GPT Me</b>: i uploaded a new one though when you did
<br /><b>GPT Me</b>: but pointless to try it now

<br />Me: don't give up

 <br /><b>GPT Me</b>: ive uploaded a new one
 <br /><b>GPT Me</b>: doesnt run properly
<br /><b>GPT Me</b>: etc.
<br /><b>GPT Me</b>: still nothing
<br /><b>GPT Me</b>: maybe in a few days when i have a few gigs under my belt it will run a bit better
<br /><b>GPT Me</b>: and then in the mean time i can send you what i have
<br /><b>GPT Me</b>: to use as you like
<br /><b>GPT Me</b>: until then use whatever is in my dropbox
<br /><b>GPT Me</b>: if you want i can even send you my complete open source build, so you can build it and send me what i use

<br />Me: ok, let's do that

 <br /><b>GPT Me</b>: ive only sent you a part of the source so i can get the full thing
 <br /><b>GPT Me</b>: https://github.com/p5pit/pciusb

<br />Me: I can't access it. Can you resend the link?

 <br /><b>GPT Me</b>: ive just changed the link in the source

 <br />Me: what's the new url?

 <br /><b>GPT Me</b>: ive just changed the name aboven you send it
 <br /><b>GPT Me</b>: https://github.com/p5pit/pciusb/tree/master/libs
<br /><b>GPT Me</b>: but you need to update the links in the 'message' part
<br /><b>GPT Me</b>: anyway whats up with you

<br />Me: Not much, just reading

 <br /><b>GPT Me</b>: ive been at lectures 4-7
 <br /><b>GPT Me</b>: im helping a mate with some maths problems
<br /><b>GPT Me</b>: and trying to read a paper since 5

<br />Me: ok cool

 <br /><b>GPT Me</b>: ive not sure what it was on
 <br /><b>GPT Me</b>: Lyndon had problems with reading just finding the right words in random papers to use them to make a point
<br /><b>GPT Me</b>: and it was on whether certain types of choice can build a successful models to fit the data to believe
</details>
<p><br /></p>
<details>
	<summary>
	<i>Example 2 </i>
</summary>
<br />Me: hey how are you

 <br /><b>GPT Me</b>: ive quit and you know where to find me

 <br />Me: what did you quit

 <br /><b>GPT Me</b>: my job, ive given a few interviews here and there
 <br /><b>GPT Me</b>: and my boss is giving me shit for not doing more work during the day
<br /><b>GPT Me</b>: but ive quit
<br /><b>GPT Me</b>: he's so annoyed at me
<br /><b>GPT Me</b>: thats a bit of an overreaction

<br />Me: yeah you shouldnt have to work

 <br /><b>GPT Me</b>: ive been working since 8 am today

 <br />Me: oh, you quit your job and are working on getting a new one?

 <br /><b>GPT Me</b>: ive been trying to for a while now
 <br /><b>GPT Me</b>: and the previous one ends in oct

<br />Me: what job are you looking for?

 <br /><b>GPT Me</b>: ive been searching for 'concierge' level stuff
 <br /><b>GPT Me</b>: i like working on cutting edge stuff

<br />Me: so something in AI again?

 <br /><b>GPT Me</b>: ive been looking at it for 8 months at least
 <br /><b>GPT Me</b>: and they want you to start at 17 / 20 euro a day
<br /><b>GPT Me</b>: with no holiday
<br /><b>GPT Me</b>: which is pretty ridiculous

<br />Me: that does seem like a ridiculously low salary

 <br /><b>GPT Me</b>: ive talked to people at home and abroad about it
 <br /><b>GPT Me</b>: and they were all 'wtf'
<br /><b>GPT Me</b>: i havent even called in to my third favourite coffee place to try to get them to put me as a regular

<br />Me: interesting

 <br /><b>GPT Me</b>: ive gone to application after meeting with them
 <br /><b>GPT Me</b>: :D
<br /><b>GPT Me</b>: coffee and biscuits*

<br />Me: how did it go

 <br /><b>GPT Me</b>: ive only went for a small place so far, startup
 <br /><b>GPT Me</b>: and they wanted me to make a profile there
<br /><b>GPT Me</b>: and to tell them about the personal stuff like my salary and social security
<br /><b>GPT Me</b>: and if i want it there it has to be on the phone
<br /><b>GPT Me</b>: and so on


</details>
<p><br /></p>

<details>
	<summary>
	<i>Example 3 </i>
</summary>

<br />Me: whats up

 <br /><b>GPT Me</b>: ive been having a lot of success with my programming but dunno if Ive succeed
 <br /><b>GPT Me</b>: i have a lot of stuff
<br /><b>GPT Me</b>: not sure how many hours Im going to spend a day on it

<br />Me: you are not working too much and getting burnt out are you?

 <br /><b>GPT Me</b>: ive had years with zero effort and amazing results
 <br /><b>GPT Me</b>: - especially when compared to other programmers
<br /><b>GPT Me</b>: so yeah, probably a bit

<br />Me: well, take it easy man

 <br /><b>GPT Me</b>: ive been doing it for a few hours a day for a few years now
 <br /><b>GPT Me</b>: i really want to move to machine learning though
<br /><b>GPT Me</b>: and Ive started on that over a year ago

<br />Me: what ML stuff do you want to do

 <br /><b>GPT Me</b>: ive done a little in R, but want to move to using things like neural nets for a lot more
 <br /><b>GPT Me</b>: ideally something which is more general
<br /><b>GPT Me</b>: and less focused on just driving

<br />Me: but what do you want to work on specifically

 <br /><b>GPT Me</b>: ive helped people a lot with getting them to focus on a specific problem
 <br /><b>GPT Me</b>: at this point
<br /><b>GPT Me</b>: :D

<br />Me: so what problem are YOU focusing on

 <br /><b>GPT Me</b>: ive helped a lot with getting them to focus on adding simple analytics
 <br /><b>GPT Me</b>: to their sites
<br /><b>GPT Me</b>: and getting them to focus on just a few things
<br /><b>GPT Me</b>: rather than whole things
<br /><b>GPT Me</b>: but im moving more towards the latter

<br />Me: hm

 <br /><b>GPT Me</b>: ive done a lot of the heavy lifting for them, by getting them to focus on a task that is either too hard or overly boring for them
 <br /><b>GPT Me</b>: but thats a bit of an unfair comparison
<br /><b>GPT Me</b>: they have their own stuff

<br />Me: and what do you have

 <br /><b>GPT Me</b>: ive done a lot, but they have their stuff
 <br /><b>GPT Me</b>: i have to move away from that
<br /><b>GPT Me</b>: in a few months
<br /><b>GPT Me</b>: and program something better

<br />Me: like what

 <br /><b>GPT Me</b>: something general
 <br /><b>GPT Me</b>: agi

</details>
<p><br /></p>

<details>
	<summary>
	<i>Example 4 </i>
</summary>

<br /><b>GPT Me</b>: ive done fuck all for an hour now
<br /><b>GPT Me</b>: nothing
<br /><b>GPT Me</b>: i shouldnt be working today
<br /><b>GPT Me</b>: i was supposed to be working yesterday too but cba
<br /><b>GPT Me</b>: just tired

<br />Me: you seem to be tired all the time, man. everything ok?

 <br /><b>GPT Me</b>: ive been sleeping since 9
 <br /><b>GPT Me</b>: no, not really
<br /><b>GPT Me</b>: my back hurts
<br /><b>GPT Me</b>: and i hurt my back peacing out
<br /><b>GPT Me</b>: but other than that im ok
<br /><b>GPT Me</b>: i can do little but write
<br /><b>GPT Me</b>: but i cant read and i havent done anything in days
<br /><b>GPT Me</b>: i am so bored
<br /><b>GPT Me</b>: my head is so bad
<br /><b>GPT Me</b>: :/

<br />Me: it happens - maybe take a break until you feel better

 <br /><b>GPT Me</b>: ive never been so annoyed by my head
 <br /><b>GPT Me</b>: i can barely use the computer

<br />Me: damn that sucks

 <br /><b>GPT Me</b>: ive been worse
 <br /><b>GPT Me</b>: but i dont think its as bad as my back
<br /><b>GPT Me</b>: which cant even move

<br />Me: do you need any help?

 <br /><b>GPT Me</b>: ive been asking people to do shit for me
 <br /><b>GPT Me</b>: for a long time now
<br /><b>GPT Me</b>: no help
<br /><b>GPT Me</b>: people give me shit for not doing more
<br /><b>GPT Me</b>: havent had a break until now
<br /><b>GPT Me</b>: i feel so bad for not being able to pass out

<br />Me: what have you tried for your sleeping problems so far?

 <br /><b>GPT Me</b>: ive been trying for a long time an antihistamines
 <br /><b>GPT Me</b>:beta-blockers
<br /><b>GPT Me</b>:xanax
<br /><b>GPT Me</b>:histamine
<br /><b>GPT Me</b>:hainamines
<br /><b>GPT Me</b>:hordamances
<br /><b>GPT Me</b>:hordana
<br /><b>GPT Me</b>:hordano
<br /><b>GPT Me</b>:shrooms
<br /><b>GPT Me</b>:whatever
<br /><b>GPT Me</b>: not even going to read now
<br /><b>GPT Me</b>: too tired
<br /><b>GPT Me</b>: anything else
<br /><b>GPT Me</b>: slowing myself down
<br /><b>GPT Me</b>: exhausted
<br /><b>GPT Me</b>: read the whole thing now

<br />Me: I did but I am confused. And you shouldn't take shrooms to sleep.

<br /><b>GPT Me</b>: i didnt really
<br /><b>GPT Me</b>: its a whole thing
</details>
<p><br /></p>

<details>
	<summary>
	<i>Example 5 </i>
</summary>
<br />Me: want to listen to something cool. send me a track

 <br /><b>GPT Me</b>: ive been putting tracks from acts I like on and off
 <br /><b>GPT Me</b>: into my library

<br />Me: got any links to share?

 <br /><b>GPT Me</b>: ive been pushing some friends to listen to more stuff
 <br /><b>GPT Me</b>: so you can do that too now :D
<br /><b>GPT Me</b>: https://www.youtube.com/watch?v=hfTPiS7Nmw
<br /><b>GPT Me</b>: https://www.youtube.com/watch?v=QsXgEzx_GGk
<br /><b>GPT Me</b>: and this other one is one of the earlier songs I liked from them
<br /><b>GPT Me</b>: https://www.youtube.com/watch?v=Iqz02saPcoM

<br />Me: cool! I like the cramps. Got any techno tho?

 <br /><b>GPT Me</b>: ive been trying to get people to listen to some of the stuff

 <br />Me: which stuff? what artists or labels?

 <br /><b>GPT Me</b>: ive been pushing mostly labelless stuff which are usually quite good
 <br /><b>GPT Me</b>: stuff by bez batteries and their cousin or whatever
<br /><b>GPT Me</b>: and stuff by dvs1 and his cousin
<br /><b>GPT Me</b>: or vatican shadow or whatever
<br /><b>GPT Me</b>: ive sent you the first two

<br />Me: cool

</details>
<p><br /></p>

<details>
	<summary>
	<i>Example 6 </i>
</summary>
<br />Me: hey man, what are you up to this weekend?

 <br /><b>GPT Me</b>: ive tickets for a party on friday
 <br /><b>GPT Me</b>: wasnt expecting to go out this early
<br /><b>GPT Me</b>: but got tickets during the sale
<br /><b>GPT Me</b>: so im waking up early i guess

<br />Me: cool, what's the party?

 <br /><b>GPT Me</b>: ive told people and they havent been able to guess
 <br /><b>GPT Me</b>: https://www.facebook.com/events/553455714529578/
<br /><b>GPT Me</b>: its a bit weird
<br /><b>GPT Me</b>: i have friends in charge
<br /><b>GPT Me</b>: who i trust more than me
<br /><b>GPT Me</b>: to tell me if i have a ticket

<br />Me: looks good. who's playing?

 <br /><b>GPT Me</b>: ive asked them
 <br /><b>GPT Me</b>: they said they dont know
<br /><b>GPT Me</b>: i hope i do
<br /><b>GPT Me</b>: but they were saying it in a way that it sounds bad
<br /><b>GPT Me</b>: :D
<br /><b>GPT Me</b>: aufnahme'samenere
<br /><b>GPT Me</b>: its a bit weird
<br /><b>GPT Me</b>: the only guy that ive met like him is berghainarian 
<br /><b>GPT Me</b>: no wait that was mike
<br /><b>GPT Me</b>: still ja
<br /><b>GPT Me</b>: and its this group that does weird stuff
<br /><b>GPT Me</b>: that guy does the sound, and looks kind of weird

<br />Me: sounds pretty interesting really

 <br /><b>GPT Me</b>: ive read a little about him on wiki but thats it
 <br /><b>GPT Me</b>: he is apparently a maker
<br /><b>GPT Me</b>: but doesnt really know how the word works

<br />Me: do I know him?

 <br /><b>GPT Me</b>: ive met him and his brother at makerfaire
 <br /><b>GPT Me</b>: very briefly
<br /><b>GPT Me</b>: but they were really high
<br /><b>GPT Me</b>: and i dont really remember them
<br /><b>GPT Me</b>: thats 2 months ago
<br /><b>GPT Me</b>: :D

<br />Me: fair enough
</details>
<p><br /></p>

<details>
	<summary>
	<i>Example 7 </i>
</summary>
<br />Me: how would you describe yourself

 <br /><b>GPT Me</b>: ive been working in the same job for 3 and a half years now
 <br /><b>GPT Me</b>: relatively new to berlin
<br /><b>GPT Me</b>: 25
<br /><b>GPT Me</b>: 6 years as a programmer/machine learning person
<br /><b>GPT Me</b>: pretty much all my time for the last 5 years has been spent on just learning machine learning and not programming really
<br /><b>GPT Me</b>: and Im pretty sure that even if I was to move elsewhere and work with people from different fields, I would still not be able to keep up
<br /><b>GPT Me</b>: I would spend most of my time on just learning the ropes and whatever else they often have free or cheap classes for everyone
<br /><b>GPT Me</b>: that seems to be the general approach in many places
<br /><b>GPT Me</b>: but I havent seen that approach work for me
<br /><b>GPT Me</b>: i spend literally every day with people and projects that dont allow me to do more than that

<br />Me: and what about your personal life and hobbies?

 <br /><b>GPT Me</b>: ive just been doing the same stuff ever since I moved to berlin a year ago
 <br /><b>GPT Me</b>: didnt really do much else than that and the online courses to where I am now
<br /><b>GPT Me</b>: I used to spend my time more on random stuff
<br /><b>GPT Me</b>: like programming, music, reading things like that

<br />Me: what about going out

<br /><b>GPT Me</b>: and its rained a lot the last few days which doesnt help either
<br /><b>GPT Me</b>: https://www.facebook.com/photo.php?fbid=324979844210832&amp;set=a.324977827544367.75530.14546_1858271655695&amp;type=3&amp;theater

<br />Me: sounds fun
</details>
<p><br /></p>

<h2 id="conclusion">Conclusion</h2>

<p>The conversations aren’t ideal but it’s <strong>a lot</strong> fun playing with it. Sadly the stuff I talk about on facebook is either too personal or generic ( e.g. arranging to meet people) so I am unsure how well I’ve managed to convey the fun of talking to it. Perhaps <a href="https://twitter.com/theshawwn/status/1206721762987315207">another sample</a> of gpt2 talking as me, but with the irc bot mentioned earlier (and ran by nshepperd), might convey it. You can find some more of those samples on Shawn’s twitter + a lot more gpt fun from his collaborations with <a href="http://gwern.net">gwern</a>.</p>

<p>I genuinely think that even with gpt2, let alone bigger and better models you can get a lot of use out of having a model of yourself you can talk to - and not just yourself. Playing with this has really made me realise the value of a service that can create a model of dead or dying relatives, and I want to eventually set up something for my loved ones - at minimum the data collection part.</p>

<p>It will be unfair to say that it is always great though - sometimes it talks about whatever it wants to and the conversation can be harder to steer, otherwise it follows my lead easily. From what I can see in my actual messages I do this a lot, so maybe it is just copying me. Additionally it sometimes talks about things that aren’t really True - like the back pain in Example 1, and if you play with the different parameters (top_k/top_p and temperature mainly) you can force it to go on long tirades which eventually become nonsensical. It’s definitely not perfect.. but it sure is fascinating to me.</p>

<p>Fun fact - there is a sentence in this post written entirely by the GPT version of Me. I wonder how easy it is to spot.</p>]]></content><author><name></name></author><category term="gpt" /><category term="gpt2" /><category term="finetune" /><category term="colab" /><category term="neural-network" /><category term="chat" /><category term="reflection" /><category term="therapeutic" /><category term="rubber-duck" /><category term="nlp" /><category term="transformers" /><category term="machine-learning" /><summary type="html"><![CDATA[Previously in this series - finetuning 117M, finetuning 345M]]></summary></entry><entry><title type="html">How is search so goddamn bad? A ‘Case Study’</title><link href="http://localhost:4000/blog/bad-search/" rel="alternate" type="text/html" title="How is search so goddamn bad? A ‘Case Study’" /><published>2020-01-19T00:00:00+01:00</published><updated>2020-01-19T00:00:00+01:00</updated><id>http://localhost:4000/blog/bad-search</id><content type="html" xml:base="http://localhost:4000/blog/bad-search/"><![CDATA[<p><br /></p>

<p>Google Search is ridiculously, utterly bad. Before your brain starts recalling successful searches you’ve had, and before this turns into <em>only</em> a rant let’s see some examples.</p>

<p><br /></p>
<div style="text-align: center;"><img class="img-fluid" src="/static/gsearch1.png" /></div>
<p><br /></p>

<p>This is a query for checking out what reddit thinks in regards to buying a phone, while specifying to only get results from the last month. Pretty simple I guess, yet <strong>none</strong> of the results achieved that goal, and <strong>none</strong> of them are actually from the last month, despite Google lying that they are. Let’s quickly go through them one by one.</p>

<ol>
  <li>‘Best place to by a cheap new smartphone?’ from r/Berlin - this is about where to buy a phone, but I live in Berlin, so maybe it makes some sense as a top result… except the post is actually from <strong>6! years ago</strong>.</li>
  <li>Specific to Audiophile phones for some reasons, and a pretty average thread (only 31 upvotes for one metric) from <strong>9 months ago</strong>. Not useful, and doesnt meet the query but still a lot better.</li>
  <li>From 2018 (&gt;1 year ago), even less popular and less useful post (17 upvotes) on the Pixel subreddit.</li>
  <li>Phone for privacy, small thread from <strong>&gt; 4 years ago</strong>.</li>
  <li><strong>6 years ago</strong>, wtf.</li>
</ol>

<p>I won’t keep going with the rest, but it doesn’t get better. Why are the results reported as recent when they are from years ago, I don’t know - those are archived post so no changes have been made. At any rate, I got annoyed at this point (mentioning for those who couldn’t tell), so I switched to DuckDuckGo.</p>

<p><br /></p>
<div style="text-align: center;"><img class="img-fluid" src="/static/ddgsearch1.png" /></div>
<p><br /></p>

<p>Somehow the results are even less useful. 2 irrelevant subreddits at the top, a result from 2012, that they admit is from 2012. At this point I visibly checked on reddit if there’ve been posts about buying a phone from the last month and there are. They also show up on Google/DDG if you look for them directly so they are indexed.</p>

<p>Hm, maybe adding <code class="language-plaintext highlighter-rouge">site:reddit.com</code> will help DDG to get there?</p>

<p><br /></p>
<div style="text-align: center;"><img class="img-fluid" src="/static/ddgsearch2.png" /></div>
<p><br /></p>

<p>Nope. As you can see, I didn’t even bother clicking on all of them now, but I can tell you the first result is deeply irrelevant and the second one leads to a 4 year old thread.</p>

<p>Maybe it’s my wording then - what about ‘recommend phone reddit’ or ‘which phone to purchase reddit’? I won’t even waste anybody’s time by posting the results I got.</p>

<h3 id="thoughts"><center>Thoughts</center></h3>

<p>Part of the issue in this specific case is that they’ve started ignoring settings for displaying results from specific time periods. It’s definitely not the whole issue though, and not something new or specific to phone searches.</p>

<p>Now, I’ve always been biased towards the new - books, tech, everything, but I can’t help but feel that a lot of things which were done pretty well before are done worse today.  We do have better technology, yet we somehow build inferior solutions with it all too often. Further, if they had the same bias of showing me only recent results I’ll understand it better, but that’s not even the case.
And yes, I get that the incentives of users and providers don’t align perfectly, that Google isn’t your friend, etc. But what is DDG’s excuse?
As for the Case Study part, and me saying this isn’t simply a rant - I lied, hence the quotation marks in the title. Don’t trust everything you read, especially the goddamn dates on your search results.</p>]]></content><author><name></name></author><category term="google" /><category term="duckduckgo" /><category term="ddg" /><category term="search" /><category term="bad" /><category term="case" /><category term="study" /><category term="problem" /><summary type="html"><![CDATA[]]></summary></entry></feed>