Combined sources blog? Merging .md files with json data?

[Julianoe]

Is it possible to compose a single feed of posts but that are coming from two different sources?
In my specific case I would have a json file coming from WordPress REST API and mardown files stored locally. Is there a way to combine the two in a chronological order for example?
I'm new to 11ty and any examples or similar project you know of would be welcome. Thanks!

[pdehaan]

Hhmm, I'm not sure what the format of the WordPress REST API response is, but I'd probably start by creating a custom collection in your .eleventy.js config file using collectionApi.getFilteredByTag( tagName ) (or getFilteredByGlob( glob )), and then merge the collection results into a single array with the WordPress data (which you might be able to do via const wpData = require("./_data/fetch-wordpress")) and then just sort the array by date (either ascending or descending order).

Biggest issue I can imagine is that the format of the 11ty collection data and exported WordPress data will be very different, so you might want to normalize the data a bit and keep the minimum amount of data like: date, title, excerpt, url, etc. so that the format of the object is consistent.

[Julianoe]

Thanks for those hints. I will research all that and see what I can do :)
As of exported data they will indeed be very limited and I will ensure they are similar when needed so that it does not cause problems

[Julianoe]

So, after a week tweaking and learning 11ty and playing around, I have a clearer idea of what I was trying to achieve when I asked the question: how do I merge the tags from WordPress posts and use them like tags in .md files?

Here are some details for reference about the process and in case people are trying to achieve something similar:

First of all to simplify the process I added a function in my WordPress theme to ease the process of getting tags info: WordPress posts contain only tags IDs and this means you have to also grab the API response for tags and match IDs with names etc... For now I went for a simpler way: I added a field in the API to output an array of tags names from WordPress:
(those are all I need as I did not edited the slugs that come from the names and the it's only info I really need anyway).

/* PHP in my wp-themes/mytheme/functions.php */
// Ajouter une nouvelle entrée dans la REST API 
// https://developer.wordpress.org/reference/functions/register_rest_field/
// https://imranhsayed.medium.com/add-custom-field-to-wordpress-rest-api-35c63c6dfff4
add_action( 'rest_api_init', 'gk_add_custom_fields' );
function gk_add_custom_fields() {
  register_rest_field(
  'post', 
  'tags_names', //New Field Name in JSON RESPONSEs
  array(
      'get_callback'    => 'gk_get_custom_fields', // custom function name 
      'update_callback' => null,
      'schema'          => null,
      )
  );
}

function gk_get_custom_fields( $object, $field_name, $request ) {
  // grab all tags for this post
  $tags_objects = get_the_tags($object['id']);
  $tags_names = [];
  // return an array of this posts' tags' names only
  foreach ($tags_objects as $tag) {
    $tags_names[] = $tag->name;
  }
  return $tags_names;
};

Then I grab the posts using a script (something similar to this) with a subtlety: I'm building two files, one is the response from the API, the other one is a simpler .json file only containing the data I want, formatted in a way similar to how 11ty organizes data (I think):

// script in grabWP.js
#!/usr/bin/env node

const fs = require("node:fs/promises");
const axios = require("axios");

grabWP();

async function grabWP() {

  // write content of fetched pages into a posts_full.json file
  const posts_full = await fetchPages();
  await fs.writeFile("_data/posts_full.json", JSON.stringify(posts_full));
  console.log(`Done! ${posts_full.length} full posts written to posts_full.json`);

  // iterate through this first file and build an array from the only info I nee:
  // (id, title, date, tag names (from a custom field created earlier in the WordPress theme, slug, content, etc.))
  const posts_simple = [];
  posts_full.forEach(post => {
    var arr = {
        "id": post.id,
        "title": post.title.rendered,
        "date": post.date,
        "tags": post.tags_names,
        "slug": post.slug,
        "content": post.content.rendered
    };
    posts_simple.push(arr);
  });
  // create another file from this data
  fs.writeFile("_data/posts.json", JSON.stringify(posts_simple));
}

async function fetchPages(page = 1) {
  const res = [];
  let totalPages = 1;
  do {
    console.log(`Fetching page ${page}/${totalPages}`);
    const { data, headers } = await axios.get("https://www.geeks-curiosity.net/wp-json/wp/v2/posts", {params: {per_page:50, page}});
    res.push(data);
    totalPages = headers['x-wp-totalpages'];
    page += 1;
  } while (page <= totalPages);
  return res.flat();
}

I end up with a nice file containing only the necessary infos from my posts from WordPress . This posts.json is used to create pages like /posts/my-newest-article-from-wordpress/ by a post.njk

// post.njk
---
pagination:
    data: posts
    size: 1
    alias: posts
permalink: "posts/{{ posts.slug }}/"
layout: default
---

{# The ids used here are from the posts.json, a simplified array of WP data generated by grabWP.js #}
<h1>{{ posts.title | safe }}</h1>

<span class="">🏷️ Tags :  
    {% for tag in posts.tags %}
    <a href="/tag/{{ tag | safe | slug }}">{{ tag | safe }}</a>
    {% endfor %}
</span>

<div class="mainContent">
    {{ posts.content | safe }}
</div>

In the same spirit I've also made a posts.njk to have a paginated list of all WP imported posts.

Now.

I'm struggling with a 11ty subtlety. How should I do to merge the tags from the posts.json and the tags from my markdown files (registered under tags: in the frontmatter of my files).
For now the tags from my WP posts don't generate tags pages while my tags from my markdown file do.

I tried changing this in many ways but to no effect, it still produces only an array of tags from markdown files, I did not manage to add the item.tags from my posts.json file

// .eleventy.js
    eleventyConfig.addCollection("tagList", function(collection) {
        let tagSet = new Set();
        collection.getAll().forEach(item => {
            // gather all tags from every note
            (item.data.tags || []).forEach(tag => tagSet.add(tag));
        });
        return filterTagList([...tagSet]);
    });

For reference here is the template tags.njk that generates the tag pages

// tags.njk
---
layout: default
pagination:
  data: collections
  size: 1
  alias: tag
permalink: /tags/{{ tag }}/
---
<h1>Étiquette: « {{ tag }} »</h1>
<ul>
{% set taglist = collections[ tag ] %}
{% for p in taglist | sort(false, false, 'data.title') %}
  <li data-tags="{{ p.data.tags }}"><a href="{{ p.url }}">{{ p.data.title }}</a></li>
{% endfor %}
</ul>

[Julianoe]

Essentially I suppose my question boils down to: how do you merge two collections into one, one data set coming from a JSON file (REST API) and the other one coming from a series of mardown files with frontmatter. Once I figure that out I can either create a common feed for merged data, and/or have 11ty grab tags and other info from both collections, etc..

Thanks in advance to anyone that can guide me in the right direction.

[pdehaan]

OK, I ran your fetch script and tried to get a basic version working locally using your WP JSON endpoint.
I think you need to add addAllPagesToCollections: true to your posts.njk pagination front matter. I couldn't figure out how to do that, so I moved the front matter stuff into a ./src/posts.11tydata.js data file with the following:

// src/posts.11tydata.js
module.exports = {
  pagination: {
    data: "posts",
    size: 1,
    alias: "post",
    addAllPagesToCollections: true,
  },
  permalink: "posts/{{ post.slug }}/",
  eleventyComputed: {
    tags: ({post}) => post.tags,
    title: ({post}) => post.title,
  },
  // layout: "default",
};

And then modified the front matter of src/posts.njk:

---
# src/posts.njk
## see ./posts.11tydata.js
---

Then I think you run into the problem where dynamically set tags don't auto-create collections, like explicitly set tags in front matter. See #1225 and #1059 for more context.

Here's my rough .eleventy.js config file:

// .eleventy.js
module.exports = function (eleventyConfig) {
  const posts = require("./src/_data/posts.json");
  // Create a tag list for our dynamically set tags via src/_data/posts.json.
  const postsTagList = posts.reduce((acc, post) => {
    (post.tags || []).forEach(tag => acc.add(tag));
    return acc;
  }, new Set());
  
  for (const t of [...postsTagList]) {
    eleventyConfig.addCollection(t, function (collection) {
      return collection.getFilteredByTag(t);
    });
  }
  ...
};

So we manually create collections based on the dynamically set tags.
Still not 100%. There are some issues w/ your tag page permalinks since they aren't really slugified:

find www/tags/**/index.html | sort -f | pbcopy

# Feels like a weird URL
www/tags/2001 l'Odyssée de l'espace/index.html
...
# Space in URL path.
www/tags/2K Games/index.html
...
# `@` in URL path.
www/tags/@Atlanta/index.html

# Note /AC/DC/ has extra folder "depth" to `/` in tag name.
www/tags/AC/DC/index.html

# More escaping stuff.
www/tags/AD&D/index.html
www/tags/adaptation jeu vidéo au cinéma/index.html
...
# Note "metal" and "Métal", when I used `|slugify` filter, these conflicted and threw permalink errors.
www/tags/metal/index.html
www/tags/Métal/index.html
...

> eleventy

[11ty] Wrote 622 files in 0.24 seconds (0.4ms each, v1.0.2)

[pdehaan]

OK, if I search posts.json" and change every "Métal" or "Metal" to lowercase "metal", then I can use the slugify filter (which is nicer than the older slug filter, in my irrelevant opinion), then it seems to build w/ much nicer permalinks. So I guess that was the only conflicting [slugified] tag name.

find www/tags/**/index.html | sort -f | pbcopy

# using `slugify` filter...

www/tags/2001-l-odyssee-de-l-espace/index.html
www/tags/2080/index.html
www/tags/2k-games/index.html
www/tags/3sonsdusoir/index.html
www/tags/48h-fp/index.html
www/tags/48hour-film-project/index.html
www/tags/8-bit/index.html
www/tags/ac-dc/index.html
www/tags/ad-and-amp-d/index.html
...
www/tags/wolf-smoke/index.html
www/tags/world-of-warcraft/index.html
www/tags/youtube/index.html
www/tags/zen-pencils/index.html
www/tags/zombillenium/index.html
www/tags/zone-retro/index.html

Versus

# using `slug` filter...

www/tags/2001-l'odyssee-de-l'espace/index.html
www/tags/2080/index.html
www/tags/2k-games/index.html
www/tags/3sonsdusoir/index.html
www/tags/48h-fp/index.html
www/tags/48hour-film-project/index.html
www/tags/8-bit/index.html
www/tags/@atlanta/index.html
www/tags/acdc/index.html
www/tags/adandampd/index.html
...
www/tags/wolf-smoke/index.html
www/tags/world-of-warcraft/index.html
www/tags/youtube/index.html
www/tags/zen-pencils/index.html
www/tags/zombillenium/index.html
www/tags/zone-retro/index.html

Or more fun w/ Bash regexp:

# using `slug` filter...
find www/tags/**/index.html | sort -f | grep -Ei "[^A-Z0-9/\.-]"

www/tags/2001-l'odyssee-de-l'espace/index.html
www/tags/@atlanta/index.html
www/tags/assassin's-creed/index.html
www/tags/baldur's-gate/index.html
www/tags/canal-+/index.html
www/tags/geek's-curiosity/index.html
www/tags/i'm-aquarius/index.html
www/tags/o'gaming/index.html

[Julianoe]

I'll look into all this when I get back on monday and come back to you. Thanks again.. Thanks for the issues linked, they seem to be aiming at similar target as I am.
Once I'm done with this tag question I'll be ready to prepare for 1st release and leave you in peace 😇

As for the conflicts and problematic dynamic tags, some I can fix from the WordPress install to prevent conflicts and confusion. Others i'll look into. I'll find a way to make all slugs to work.

[pdehaan]

If you don't care too much about the dynamic tags (ie: you aren't displaying them), you could always try slugifying them before adding them to the Set:

// .eleventy.js
 const posts = require("./src/_data/posts.json");

 module.exports = function (eleventyConfig) {
  const slugify = eleventyConfig.getFilter("slugify");
 
  const postsTagList = posts.reduce((acc, post) => {
    (post.tags || []).forEach(tag => acc.add(slugify(tag)));
    return acc;
  }, new Set());

Although then you might also need to slugify them when you set them via eleventyComputed as well:

// src/posts.11tydata.js (snippet)
  eleventyComputed: {
    tags(data) {
      return data.post.tags.map(tag => this.slugify(tag));
    },
    title: (data) => data.post.title,
  },

Would take a bit of tweaking to get just right, but if there is a eleventyComputed.getFilter("slugify") API that you can use depending on your context. Or in eleventyComputed it might be in the data context, or require you to use non-arrow functions.

[Julianoe]

Looking into your Merged collection repo you seem to say that it's not a good idea to proceed in this way.
Maybe am I looking at this wrong but I'm not really sure how else we could do this. Is there another more straight forward way to have tags in common for pages/posts from 2 different sources (json file + markdown notes)? I thought that would need to mean merging the collections but maybe it does not.

It'd be a great way to tie a web of links between those two content types and avoid them being in 2 different tagging silos.

[pdehaan]

I think the eleventyComputed.addCollection("whatever", collectionApi => collectionApi) trick might be risky and could break at any point in the future if @zachleat changes collections plumbing. It definitely doesn't feel like how one should use the API. But it worked for my limited testing, and was easier than getting a list of all unique, dynamic tags from the WordPress data source and recreating each collection in the .eleventy.js config file. But doing something like https://github.com/pdehaan/11ty-2552-merged/blob/main/src/posts/pagination.11tydata.js let me merge local files w/ [mock] fetched WordPress results and put them all into a collections.notes API. Probably still has a lot of plumbing left to add tag archive pages and pagination or whatever else.

I also got it working w/ LiquidJS templates in https://github.com/pdehaan/11ty-dynamic-collection-liquid, but needed to add custom filters for each of the collectionApi, TemplateCollection, methods.