Proposal: Plug-ins v2 and Thin Buffalo Binary

[markbates]

The current plug-in system for Buffalo is slow, error prone, and confusing for many users. This is a proposal for a alternative approach to plug-ins that, hopefully, addresses these issues.

As well as proposing a major revision of plug-ins, this proposal also introduces the concept of a "thin" Buffalo binary.

In this proposal I will use github.com/gobuffalo/pop as a library, package, and tool, that some people may not want to use.

The Fat Binary

Let's start with the buffalo binary. It is a "fat" binary. This means that it contains everything, from the web-server, to the generators, etc... Because of this several problems arise. The first is that this "fat" binary contains a lot of dependencies, regardless of whether the end user will ever use those features.

The current binary has a hard dependency on pop. The buffalo generate resource command, for example, generates pop style resources. So if the app in question uses another ORM, or none at all, or if they only generate JSON, then this dependency shouldn't be present.

Another problem this "fat" binary presents is that it can often be a version mismatch with the app it is being run against. If the app is v0.12.x but the binary is v0.15.0, then the results of running buffalo generate resource, for example, wouldn't be guaranteed to be consistent.

The Plug-in Problem.

Buffalo plugins have a set of rules that must be followed for them to be consider, by Buffalo, as a plugin.

• Plug-ins must be named in the format of buffalo-<plugin-name>. For example, buffalo-myplugin.
• Plug-ins must be executable and must be available in one of the following places:
  • in the $BUFFALO_PLUGIN_PATH
  • if not set, $GOPATH/bin, is tried
  • in the ./plugins folder of your Buffalo application
• Plug-ins must implement an available command that prints a JSON response listing the available commands.

When a command is run, such as buffalo db migrate, Buffalo will try and find the plug-in (binary) and if found, it will then try and shell out the command of migrate to it.

This constant shelling out and searching for executables is error prone, confusing, slow, and again, has similar versioning concerns to that of the buffalo binary itself.

Thin Binary, Fat Binary

To solve these problems, I suggest the following.

First, we make the buffalo binary a "thin" one. We remove anything needed for running inside of an application. For example, buffalo dev is not required outside of an application, so why is it needed in the binary?

The buffalo binary should only contain what it needs when running its "outside" duties, such as buffalo new.

Instead, I propose using a similar approach to github.com/markbates/grift.

The "thin" buffalo binary when a command such as buffalo db migrate would [hands wave wildly in the air] compile a "fat" buffalo specifically for the current application, using the versions of Buffalo, and other libraries declared in the go.mod.

The pop plugin would be added to the application via a standard import.

// <app>/plugins/plugins.go
package plugins

import _ "github.com/gobuffalo/buffalo-pop"

The buffalo-pop plugin would register any sub-commands it may have for the new "fat" binary that is built.

Other Changes

With these changes the buffalo-plugins.toml file will no longer be needed, as well as all of the buffalo plugin sub-commands.

Sub-commands would be forced under their plug-in's name. So if the buffalo-pop plugin is registered with the name "pop" then the buffalo db migrate command would become buffalo pop migrate. Gone, also, would be "wrapping" another command.

Interfaces would be used to allow for simplier access points into the plug-in system. For example, we might have interfaces for such things as listing any commands, or implementing a "fix" command for things like buffalo fix to hook into.

// Not final interfaces, just rough ideas.

type Plugin interface {
  Name() string
}

type Fixable interface {
  // optional interface for hooking into "fix" sub-command
  // buffalo fix (will also call buffalo-pop/plugin.Plugin#Fix - if implemented)
  Fix(context.Context) error
}

type Commanding interface {
  Commands() (Commands, error)
}

Conclusion

I would love to hear community feedback on this proposal. Also, if there is anyone out there that would want to take lead on such a proposal, please speak up. :)

[dnnrly]

Nice, that makes a lot of sense! I can see this solving PATH related problems well.

[timuckun]

Here is a crazy idea.

Why not have a fat buffalo for development and a thin one for production? Maybe they could be managed with symlinks or a shim shell script.

[hubyhuby]

Food for thoughts from a humble user of buffalo.
Discussing with friends using Ruby on rail or Laravel, the discussion ends always at one point on how "plugins-gems-modules" makes them sooo productive.
Most importantly the plugins are well supported & updated; therefore previously generated apps can be MAINTAINED easily.

On the long run , the success of buffalo will depend in part on those plugins, their number, quality, and how easy they are to maintain.

I am not expert in the guts of the Buffalo BIN, but the proposal seems to go in the way of an easier maintenance of plugins; which is great.
Things like being able to launch a project in 0.12.x with a buffalo Bin in 0.15.x would be great.

[markbates]

I've written a small POC around this, for those that are interested: https://github.com/markbates/bluffalo

/*
This cli package would live in the bluffalo application.
*/
package cli

import (
	"context"

	"github.com/markbates/bluffalo"
	"github.com/markbates/bluffalo/fauxplugs/goth"
	"github.com/markbates/bluffalo/fauxplugs/heroku"
	"github.com/markbates/bluffalo/fauxplugs/plush"
	"github.com/markbates/bluffalo/fauxplugs/pop"
)

// Main is the entry point for the bluffalo application
// this is what will be called by main.go
// It would be used by tools like `bluffalo dev` and
// `bluffalo build`.
func Main(ctx context.Context, args []string) error {
	// app := actions.App()
	// if err := app.Serve(); err != nil {
	// 		return err
	// }
	return nil
}

// Bluffalo is the entry point for the `bluffalo` binary.
// It allows for registering plugins to enhance the `bluffalo` binary.
// 	bluffalo generate -h
// 	bluffalo generate pop ...
// 	bluffalo fix
// 	bluffalo fix plush ...
// 	bluffalo fix -h
func Bluffalo(ctx context.Context, args []string) error {
	b, err := bluffalo.New(ctx)
	if err != nil {
		return err
	}

	b.Plugins = append(b.Plugins,
		pop.New(),
		goth.New(),
		heroku.New(),
		plush.New(),
	)

	return b.Main(ctx, args)
}

Plugins would implement different interfaces for the different parts of the Buffalo binary that it wants to interact with, such as buffalo fix and buffalo generate.

// Plugin is the most basic interface a plugin can implement.
type Plugin interface {
	// Name is the name of the plugin.
	// This will also be used for the cli sub-command
	// 	"pop" | "heroku" | "auth" | etc...
	Name() string
}

// Fixer is an optional interface a plugin can implement
// to be run with `bluffalo fix`. This should update the application
// to the current version of the plugin.
// The expectation is fixing of only one major revision.
type Fixer interface {
	Fix(ctx context.Context, args []string) error
}

// Fixer is an optional interface a plugin can implement
// to be run with `bluffalo fix`
type Generator interface {
	Generate(ctx context.Context, args []string) error
}

Further interfaces could be added to allow a plugin to provide the templates, for example, during resource generation, while another generates the model portion.

Once installed the bluffalo binary has little that it can do. Commands, such as bluffalo new would live is this binary, the rest of the commands will be sent through the coke/cli.Bluffalo function, which creates a new Bluffalo cli, appends the app's desired plugins, and then runs it.

go install ./cmd/bluffalo

cd coke

bluffalo -h

bluffalo fix -h
bluffalo fix
bluffalo fix pop
bluffalo fix plush

bluffalo generate -h
bluffalo generate goth facebook twitter

[duckbrain]

I'd like to suggest a modification to the plugin file format that could really help CI environments and ensure the buffalo tool is versioned with go modules. This could also enable configuring plugins for a specific project by running some code before calling cli.Run().

// <app>/plugins/plugins.go
package main

import _ "github.com/gobuffalo/buffalo/cli"
import _ "github.com/gobuffalo/buffalo-pop"

func main() {
   cli.Run()
}

The installed buffalo tool would effectively become a smart alias for go run ./plugins. (It could cache a compiled binary.) In environments where buffalo isn't installed, The go run command could be substituted.

Other than that the buffalo tool could still have buffalo new, which can also be abstracted to simple go commands. Maybe we'd like buffalo new -v 0.16.0 to allow you to specify the version of buffalo to use. It can shell out to something like the following.

go mod init github.com/duckbrain/coolproject
(
    cd coolproject
    go get github.com/gobuffalo/buffalo@v0.16.0
    go run github.com/gobuffalo/buffalo/cli/initialize-project
)

Then the buffalo tool could support future versions of Buffalo too.

To clarify, in these hypotheticals, the github.com/gobuffalo/buffalo project would have the following packages.

• buffalo/ The current CLI tool that becomes a wrapper calling go commands. (main package)
• cli/ New package that implements the commands and API for plugins
• cli/initialize-project/ Package to run to initialize a new project (main package) (create the files, add modules, run yarn, etc.)

GQLGen uses this technique to a lesser degree, and we've been using go run for "scripts" in our project with great success.

Edit: Apparently I wasn't reading the POC comment or the sample project, the version controlled tool looks fantastic. I'm looking forward to this.