File watcher to speed up development!

[ldiebold]

Hi all!

I wrote some typescript code that creates a "watcher" for micropython (only tested on linux).
Anything changes to files within ./src are automatically sent to the device.

There's a few caveats. For example, you'll likely want to upload all files to begin with as there's no "initial sync" script yet, and it won't work while running a repl (due to how micropython/mpremote work). We could definitely extend this to work around those problems though!

import * as chokidar from 'chokidar'
import { exec } from 'child_process'
import * as path from 'path'
import * as readline from 'readline'

const srcDir = path.resolve('./src')
const deviceDir = '/'

const watcher = chokidar.watch(srcDir, {
  ignored: /(^|[\/\\])\../, // ignore dotfiles
  persistent: true,
})

const makeDirOnDevice = (folderPath: string): Promise<void> => {
  return new Promise((resolve, reject) => {
    const mkdirCommand = `mpremote mkdir ":${folderPath}"`;
    exec(mkdirCommand, (err, stdout, stderr) => {
      if (err && !stderr.includes("File exists")) {
        resolve();
        return;
      }

      if(stderr) {
        reject(`error making folder '${folderPath}': ${stderr}`)
      }

      if(err) {
        reject(`error making folder '${folderPath}': ${err}`)
      }
    });
  });
};

watcher.on('change', async (filePath) => {
  if (filePath.endsWith('.tmp')) {
    return
  }
  const relativePath = path.relative(srcDir, filePath)
  const devicePath = path.join(deviceDir, relativePath)
  const command = `mpremote cp "${filePath}" ":${devicePath}"`
  const folderPath = path.dirname(devicePath);

  await makeDirOnDevice(folderPath)

  await exec(command, async (err, stdout, stderr) => {
      console.log(`'${relativePath}' updated`)
      if (err) {
        console.error(`Error: ${err}`)
        return
      }
    if (['main.py', 'boot.py'].includes(relativePath)) {
      console.log(`'${relativePath}' changed, resetting the device`)
      await exec('mpremote reset', (err, stdout, stderr) => {
        if (err) {
          console.log('got an error')
          console.error(`Error: ${stderr}`);
          return;
        }
        console.log('Device reset');
      });
    }
  })
})

readline.emitKeypressEvents(process.stdin)
process.stdin.setRawMode(true)
console.log('Press Ctrl+R to hard reset the device')

process.stdin.on('keypress', (str, key) => {
  if (key.ctrl && key.name === 'r') {
    // Add your custom command here
    const command = 'mpremote reset'
    exec(command, (err, stdout, stderr) => {
      if (err) {
        console.error(`Error: ${stderr}`)
        return
      }
      console.log('hard resetting the device')
    })
  }
  if (key.ctrl && key.name === 'c') {
    process.exit()
  }
})

To run it, you'll also need a package.json and to install the dependencies (npm, pnpm, yarn, bun etc)

package.json

{
  "name": "file-watcher",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "dev": "bun ./watch.ts"
  },
  "keywords": [],
  "author": "Luke Diebold",
  "license": "ISC",
  "dependencies": {
    "@types/node": "^20.8.7",
    "chokidar": "^3.5.3",
    "readline": "^1.3.0"
  },
  "devDependencies": {
    "bun": "^1.0.7"
  }
}

Does something like this exist? If not... any thoughts on if this is worth developing furthur (deleting files, renaming folders etc)?

[jimmo]

Does something like this exist? If not... any thoughts on if this is worth developing furthur (deleting files, renaming folders etc)?

@ldiebold I have seen a few other things like this, but FYI have you seen mpremote mount ?

Also, speaking of mpremote, in #11777 I made recursive sync do hash comparisons which makes copying a whole directory tree much faster. Adding a --watch mode to this might be useful.

[ldiebold]

Thanks for the reply. (and from a fellow aussie too!)

mpremote mount is awesome! I stopped using it because I'm regularly restarting the device, and updating main/boot.
To my understanding, when using mount it still uses main.py and boot.py on the device, because the folder is mounted after those two are run.

Is this the case, or am I missing something here? It seems like mpremote mount would be a much better dev experience!

Come to think of it, maybe I need to start moving my code away from main/boot as the project grows.

[andrewleech]

Yes boot and main are run at startup before mpremote can get involved.

I personally find it's always best to keep main.py essentially empty other than calling out to my application module to start the real code, eg.

import app
app.run()

When doing "live development" with mpremote I'll then disable the run line from main and instead do something like

mpremote mount . repl --inject-code "import app;app.run()"

Then every time I make a change locally I switch to terminal and hit:

• Ctrl-c to interrupt old previous run
• Ctrl-d to soft reset / remount
• Ctrl-j to rerun the injected run snippet

[ldiebold]

This helps a lot, thankyou @andrewleech!

[jimmo]

@andrewleech @ldiebold

This pattern of using mount with repl --inject-code is in the docs but I really should break it out into its own section because it's really worth knowing.

One slight modification based on the way I use it -- add a \n to the end, i.e. --inject code "import app;app.run()\n" so that the snippet runs immediately without having to press return :) (this isn't in the docs and I should add it)

[peterhinch]

Interesting, and very neat. The pattern I've been using is

$ mpremote mount . exec "import app; app.run()"

I guess the beauty of --inject-code is that you have access to the REPL between code iterations.