elm-pages 2.0 routing RFC

[dillonkearns]

Summary

The current prototype uses a file-based router. Some sources of inspiration:

• elm-spa (and in particular the routing from the upcoming release [permalink])
• Routing in NextJS
• Routing in NuxtJS

Module identifier limitations in Elm

NextJS uses the following conventions

1. pages/post/create.js - Will match /post/create
2. pages/post/[pid].js - Will match /post/1, /post/abc, etc. But not /post/create
3. pages/post/[...slug].js - Will match /post/1/2, /post/a/b/c, etc. But not /post/create, /post/abc
4. pages/post/[[...slug]].js will match /post, /post/a, /post/a/b, and so on.

The main difference between catch all and optional catch all routes is that with optional, the route without the parameter is also matched (/post in the example above).

The terms they use for those are:

1. Route with no dynamic segments
2. Route with a dynamic segment (pid)
3. Route with catch all
4. Route with optional catch all

A naming convention for elm-pages

With elm-pages, we get nicely typed RouteParams based on which of these we use. We have to use a different naming convention because Elm modules must start with a capital letter, followed by alphanumeric or _ characters.

So these are the rules I'm imagining for elm-pages' file-based routing:

• Camel-casing is converted to kebab casing (elm-spa uses that same convention)
• A route segment with a single trailing _ is a dynamic segment
• A route segment with two trailing _'s is an optional dynamic segment
• If a dynamic segment is named REST (all-uppercase), then it matches a list of the rest parameters (all segments after)

Module name	Matching routes	RouteParams type
Page.Post.Slug_	/post/a, but not /post	{ slug : String }
Page.Post.Slug__	/post/a, AND /post	{ slug : Maybe String }
Page.Post.REST_	/post/a/b/c, but not /post	{ slug : NonEmptyList String }
Page.Post.REST__	/post/a/b/c AND /post	{ slug : Maybe (NonEmptyList String) }

Route Precedence

Unparameterized Routes will always match first. So between Page.Post.Slug_ and Page.Post.Create, the create page has no route params so it will always match and Slug_ will be overshadowed (not match) for that specific path.

Similarly, I believe it makes sense for catch all routes to always match after a regular dynamic segment. So for example, for the path /post/tag/elm, Page.Post.Tag_.TagName_ would match, and Page.Post.REST_ would be overshadowed (not match).

Index routes

In the current 2.0 alpha, the special route name Index is treated like index.html is in web page routing.

So Page.Index is the root route (/). You can also currently do Page.Post.Index, which would match /post.

In the latter case, this is redundant (you could simply use the module name Page.Post).

Another way to approach this would be to use the top-level module name Page for this purpose. There is an elegance to this approach. The problem is that I'm currently using the module name Page for the API for defining a Page Module. For example,

module Page.Blog exposing (page, Msg, Model)

import Page exposing (Page)

page : Page
page =
    Page.prerenderedRoute { data = ... }

This is a nice, simple name. But maybe we could come up with a different name so that we could eliminate the special rules for index routes and instead just use Page as the module name for the root route.

RouteParam field normalization

If RouteParams has a field which cannot be turned directly into a valid Elm field name, it can either be normalized, or the framework could give an error.

For example

Page.Post.Type_

Since type is a reserved word in Elm, then the field name would have to be modified to be valid:

type alias RouteParams = { type_ : String }

Adding an underscore in these cases is sufficient for reserved words. I think it makes sense to just normalize them, since each route parameter should have a unique name so we shouldn't need to worry about collisions.

Leading numbers or special characters will result in an invalid Elm module name, so the Elm compiler will catch that problem before the framework has to deal with it.

[dillonkearns]

For reference, I'll also share an experiment I explored which resulted in a dead end.

Url Parser-based routing

So I've been pondering this some more, and there's a design dead end that I'm not finding a way out of with the Url Parser-based routing.

You can't have a Route custom type that the user can access from their Page Templates. Because the type Route = Blog__Slug_ Blog__Slug_.RouteParams | ... is auto-generated, and points to the type in the Page Template. It would cause a circular reference.

I think having access to the Route type will be a cool feature being able to link and navigate to pages. So this would be an unfortunate limitation. It also may be important to be able to give a List Route in certain places during the build, for example when generating files so you could generate a sitemap.

With the Url Parser-based types, you would have to evaluate Elm code to figure out the types.
What if the Elm code isn't in a compiling state?
What would you put there?
It gets super messy and hacky because of that.
Not to mention, what about a case like this:

page : Page MyCustomRouteType ...

type MyOpaqueRouteType = Thing1 Int | Thing2 String

Hardcoding the route there is impossible. You'd have to refer to the type in that module because it's defined in the module itself. So no getting around the circular dependencies in that instance.

The original conversation is at https://discord.com/channels/534524278847045633/727578237101932556/830632281869254667.

[strickinato]

👋

Use case:

I'd love to clarify my understanding against the existing example on the site - featuring "Sibelius"

Right now:

You might define:

src/Template/Composer.elm

You'd create

content/composer/sibelius.md with the frontmatter:

template: composer
composer: sibelius

And then in the decoder for the Metadata, you'd match the 'composervalue oftemplateto the type exported fromTemplate/Composer, and then when a user visited mysite.com/composers/sibelius` - they'd be brought there.

This works well and great, and ALL pages are generated ahead of time.

But it falls short - because you would need to definite a content/composers/*.md for ALL COMPOSERS on the site.

This RFC Is shooting for:

Let's imagine I had access to an API of composer information, right now, there's now way to wrap that api with pages. So in the world, I might:

create Page/Composer/Name_.elm

Now I don't need to have anthing inside of a content/composer directory. I can simply define that page. Then, when someone visits mysite.com/composer/sibelius, I load the page with a { name : String } - where String is "sibelius" - and I can fetch whatever information I want from the API.

Is that right?

Assuming I understand properly, a question I have is:

• How can I define which routes get for generating the site at build time? That is, in the second example above, (where I don't have a content/composer/sibelius.md), how would I tell elm-pages to build the pages /composer/sibelius and. /composer/mozart

A "feature request"

Leading numbers or special characters will result in an invalid Elm module name, so the Elm compiler will catch that problem before the framework has to deal with it.

I ran into this when converting to elm-pages from another framework - which expected the files to be prefixed with the date of posting - ie content/composers/2021-04-28-sibelius.md

This is kind of nice for the purposes of both filesystem alphabetization and organization AND compatibility with other frameworks. I think it would be great if elm-pages simply dropped numbers and special characters rather than exploding when hitting them.

[dillonkearns]

Hey! Good question. The API for telling which routes to pre-render has DataSource access built-in (formerly known as StaticHttp, but now called DataSource in 2.0 to reflect the fact that it also includes reading files, not just HTTP data).

The example you're describing would look like this:

module Page.Composer.Name_ exposing (Data, Msg, Model, page)

type alias RouteParams = { name : String }

type alias Data = ()

page : Page RouteParams Data
page =
    Page.prerenderedRoute
        { data = data
        , head = head
        , routes = routes
        }
        |> Page.buildNoState { view = view }


routes : DataSource.DataSource (List RouteParams)
routes =
    DataSource.succeed [ { name = "sibeilus" } ]

data : RouteParams -> DataSource.DataSource Data
data route =
  DataSource.succeed ()

As far as elm-pages is concerned, it doesn't matter where a DataSource came from, only what Elm type (and specific value) you end up with. So the cool thing is, the example I gave for brevity is DataSource.succeed [...]. But it could just as well be DataSource.Http.get ..., or DataSource.File.read ..., etc. It can even do a Glob to get a list of files matching a pattern.

So you can do whatever you need to from those building blocks!

There's also no more concept of built-in Metadata. Instead, you can just use the File DataSource to decode frontmatter if you need that for your page. Same idea for what used to be the rendered body. Now you just pull in a data source, if you happen to need it from a page. So there's no more coupling to the content/ folder.

Regarding the 2021-04-28-sibeilus routing part, I've also entirely removed the generated Pages.pages record, so that problem is gone. What we have now in 2.0 is a Route module. So you have a generated type like this:

module Route exposing (urlToRoute, Route(..), routeToPath)

type Route = Composer__Name_ { name : String } | ...

I think you'll really enjoy those features in 2.0 😄

[dillonkearns]

For reference, here are some working examples in the current alpha:

Glob API example (gets all blog post markdown file paths and their slugs):

elm-pages/examples/docs/src/Article.elm

Lines 20 to 27 in e1b4d90

	blogPostsGlob : DataSource.DataSource (List { filePath : String, slug : String })
	blogPostsGlob =
	Glob.succeed BlogPost
	|> Glob.capture Glob.fullFilePath
	|> Glob.ignore (Glob.literal "content/blog/")
	|> Glob.capture Glob.wildcard
	|> Glob.ignore (Glob.literal ".md")
	|> Glob.toDataSource

Defining the routes and data for Page.Blog.Slug_:

elm-pages/examples/docs/src/Page/Blog/Slug_.elm

Lines 49 to 56 in 2b78dd8

	page : Page RouteParams Data
	page =
	Page.prerenderedRoute
	{ data = data
	, head = head
	, routes = routes
	}
	|> Page.buildNoState { view = view }

The Data for that page is getting the frontmatter and running a decoder to turn it into the equivalent of Metadata before (but now you just pull it in if and when you want it, not forced to for each page):

elm-pages/examples/docs/src/Page/Blog/Slug_.elm

Lines 184 to 201 in 2b78dd8

	data : RouteParams -> DataSource.DataSource Data
	data route =
	StaticFile.request
	("content/blog/" ++ route.slug ++ ".md")
	(OptimizedDecoder.map2 Data
	(StaticFile.body
	|> OptimizedDecoder.andThen
	(\rawBody ->
	case rawBody |> MarkdownRenderer.view |> Result.map Tuple.second of
	Ok renderedBody ->
	OptimizedDecoder.succeed renderedBody
	Err error ->
	OptimizedDecoder.fail error
	)
	)
	(StaticFile.frontmatter frontmatterDecoder)
	)

And the overall flow, as you hinted at, is

• Tell me which routes to prerender for a given Page Module (sibelius, mozart, etc.) - you can pull in DataSource data for that
• Then, if I give you RouteParams, tell me what data that given route needs (often a DataSource based on those RouteParams, like reading composers/mozart.md, etc.

[MarcoDaniels]

Hey.

Based on the Module name and matching routes, will it also provide support to create "root" routes like /about-us or /not-found?

The use case would be when all content for the application lives in a CMS including page urls, it would be nice to be able to map all content to pages, therefore having support for something like:

Module name	Matching routes	RouteParams type
Page.Slug_	/a	{ slug : String }
Page.REST__	/a/b/c	{ slug : Maybe (NonEmptyList String) }

I believe this could create a potential conflict in the case of someone creating a / page and it would be consider index.html.

[dillonkearns]

Absolutely, top-level routes work just as any other dynamic routes, just that they're subject to the same route precedence rules that I described above.

So if you have a Page.AboutUs module, then that will take precedence and Page.Slug_ will not match for about-us. If you don't have Page.AboutUs defined, then the parameterized route Page.Slug_ would match.

Or as your last point there describes, you could also use a REST__ route to handle every single route, and not define any specific pages, and let all of your routes be defined dynamically from a CMS (or any other DataSource) 👍

[georgesboris]

I feel really good about this design and I'm looking forward for more details regarding the DataSources.

My comment here is almost a nitpick – why use REST_ instead of something like ANY_ or ALL_ ? I asked because the word REST also means, well… REST, and there are other frameworks that autogenerate REST routes (like Rails), I know it doesn't necessarily make sense on an elm-pages app but I do think it causes some friction that other names wouldn't.

Also – I really like the fact that Page.Index is its own thing (and that I can have all my pages inside a Page/ folder, even then homepage) and Page represents a set of utilities regarding all pages. Just my two cents. 🙂

[dillonkearns]

Thanks for sharing your thoughts here Georges!

I considered that ambiguity with the word REST as well.

The terms I've been considering at the top of my list are:

1. REST_
2. ALL_
3. SPLAT_

The thing that I don't like so much about ALL_ is that it's misleading in that it's not giving you all of the segments, just whatever is left (the rest of the segments).

In a lot of languages, that concept is called Rest Parameters. So that was my thinking behind leaning towards REST_.

But I do agree that the concept of REST routes/APIs is an unfortunate ambiguity there.

SPLAT_ seemed a bit verbose and jargony, which is why I wasn't leaning towards that option. I wonder if there are any other good options or any other prior art around naming choices other frameworks have made here.

I know that NextJS refers to these as "catch all routes". But CATCHALL_ or CATCH_ don't seem very clear.

[dillonkearns]

Alright, I went with SPLAT_. So this is now implemented. What do you think?

No route params

elm-pages/examples/routing/src/Page/Slide.elm

Lines 20 to 21 in 5365e4b

	type alias RouteParams =
	{}

Dynamic route param

elm-pages/examples/routing/src/Page/Slide/Number_.elm

Lines 32 to 33 in 5365e4b

	type alias RouteParams =
	{ number : String }

Required splat param

elm-pages/examples/routing/src/Page/Date/SPLAT_.elm

Lines 22 to 23 in f2a212f

	type alias RouteParams =
	{ splat : ( String, List String ) }

Optional splat

elm-pages/examples/routing/src/Page/SPLAT__.elm

Lines 22 to 23 in a58dfd3

	type alias RouteParams =
	{ splat : List String }

Note that the optional splat in this case is at the top-level, which means it will be a catch all for any routes that aren't overshadowed by any of the above route types.

Some test cases

For reference, I have some automated tests that run against the generated code here:

elm-pages/examples/routing/tests/RouteTest.elm

Lines 31 to 82 in 709d11e

	, fuzz routeFuzzer "reversible" <|
	\route ->
	let
	asUrl =
	Just route
	|> Route.routeToPath
	|> (\path ->
	{ path = path |> String.join "/" }
	)
	asRoute =
	asUrl
	|> Route.urlToRoute
	in
	{ route = asRoute
	, path = asUrl.path
	}
	|> Expect.equal
	{ route = Just route
	, path = asUrl.path
	}
	]
	routeFuzzer : Fuzzer Route
	routeFuzzer =
	Fuzz.oneOf
	[ Fuzz.map
	(\string -> Route.Cats__Name__ { name = Just string })
	nonEmptyUrlEscapedString
	, Fuzz.int
	|> Fuzz.map
	(\number ->
	Route.Slide__Number_ { number = String.fromInt number }
	)
	, Fuzz.int
	|> Fuzz.map
	(\number ->
	Route.Date__SPLAT_ { splat = ( String.fromInt number, [] ) }
	)
	, Fuzz.map2
	(\first rest ->
	Route.Date__SPLAT_
	{ splat =
	( String.fromInt first
	, List.map String.fromInt rest
	)
	}
	)
	Fuzz.int
	(Fuzz.list Fuzz.int)
	]

As well as this test for the underlying route sorting function here:

https://github.com/dillonkearns/elm-pages/blob/709d11ef6bf46d38c84f523e75b78d8752dba55f/generator/test/route-sorting.js