Add CI/CD

[carlgieringer]

Continuous integration/continuous deployment. Run linting, run tests. Deploy each of the following, if changes.

• web app
• API
• Add CI/CD for our iOS mobile app #284
• Add CI/CD for Android mobile app #285
• Add CI/CD for Chrome extension #286
• Add CI/CD for terraform #355
• Add CI/CD for new-style lambdas #486

[carlgieringer]

Let's create a github action like:

yarn run check-preconditions:all
yarn run lint:all
yarn run typecheck:all
yarn run test:all

[carlgieringer]

※ yarn workspaces foreach -Apv run check-preconditions
Type Error: e.every is not a function
    at test (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:10:36972)
    at A.validateAndExecute (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:337:527)
    at async oo.run (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:351:1846)
    at async oo.runExit (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:351:2013)
    at async i (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:448:12377)
    at async r (/Users/transverna/code/github/Howdju/howdju/.yarn/releases/yarn-3.1.0.cjs:448:10617)

Fixed with:

※ yarn plugin import workspace-tools
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/@yarnpkg/cli/3.1.0/packages/plugin-workspace-tools/bin/%40yarnpkg/plugin-workspace-tools.js
➤ YN0000: Saving the new plugin in .yarn/plugins/@yarnpkg/plugin-workspace-tools.cjs
➤ YN0000: Done in 6s 790ms

[carlgieringer]

howdju-common preconditions failing:

※ yarn workspaces foreach -Apv run check-preconditions
➤ YN0000: [howdju-common]: Process started
➤ YN0000: [premiser-api]: Process started
➤ YN0000: [premiser-ui]: Process started
➤ YN0000: [howdju-common]: /Users/transverna/code/github/Howdju/howdju/howdju-common/bin/check-preconditions.js:21
➤ YN0000: [howdju-common]:     throw new Error("file system standalone AJV does not match generated one." +
➤ YN0000: [howdju-common]:     ^
➤ YN0000: [howdju-common]:
➤ YN0000: [howdju-common]: Error: file system standalone AJV does not match generated one. Please run gen-standalone-schema-validation and commit the result.
➤ YN0000: [howdju-common]:     at checkStandaloneSchemaValidationCode (/Users/transverna/code/github/Howdju/howdju/howdju-common/bin/check-preconditions.js:21:11)
➤ YN0000: [howdju-common]:     at Object.<anonymous> (/Users/transverna/code/github/Howdju/howdju/howdju-common/bin/check-preconditions.js:1:1)
➤ YN0000: [howdju-common]:     at Module._compile (internal/modules/cjs/loader.js:1063:30)
➤ YN0000: [howdju-common]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
➤ YN0000: [howdju-common]:     at Module.load (internal/modules/cjs/loader.js:928:32)
➤ YN0000: [howdju-common]:     at Function.Module._load (internal/modules/cjs/loader.js:769:14)
➤ YN0000: [howdju-common]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
➤ YN0000: [howdju-common]:     at internal/main/run_main_module.js:17:47
➤ YN0000: [howdju-common]: Process exited (exit code 1), completed in 1s 850ms
➤ YN0000: [premiser-ui]: Process exited (exit code 0), completed in 2s 388ms
➤ YN0000: [premiser-api]: From github.com:Howdju/howdju
➤ YN0000: [premiser-api]:  * [new branch]      dependabot/npm_and_yarn/moment-2.29.4 -> origin/dependabot/npm_and_yarn/moment-2.29.4
➤ YN0000: [premiser-api]: Process exited (exit code 0), completed in 5s 588ms
➤ YN0000: Done in 5s 610ms

[carlgieringer]

Getting tests working in howdju-mobile-app is hard.

Initially I get an error trying to load an image:

※ yarn run test
 FAIL  __tests__/App-test.tsx
  ● Test suite failed to run

    Invalid or unexpected token

      Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.
      Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.
      By default "node_modules" folder is ignored by transformers.
      Here's what you can do:
       • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
       • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
       • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
       • If you need a custom transformation specify a "transform" option in your config.
       • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.
      You'll find more details and examples of these config options in the docs:
      https://jestjs.io/docs/configuration
      For information about custom transformations, see:
      https://jestjs.io/docs/code-transformation
      Details:
      /Users/transverna/code/github/Howdju/howdju/howdju-mobile-app/node_modules/@react-navigation/elements/lib/commonjs/assets/back-icon.png:1
      ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){�PNG

      SyntaxError: Invalid or unexpected token

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        6.697 s
Ran all test suites.

I think this is fixed by https://jestjs.io/docs/webpack#handling-static-assets.

Then I get:

※ yarn run test
 FAIL  __tests__/App-test.tsx
  ● Test suite failed to run

    Cannot find module 'react-native' from '../../react-native-share-menu/index.js'

    Require stack:
      /Users/transverna/code/github/Howdju/react-native-share-menu/index.js
      src/App.tsx
      __tests__/App-test.tsx


      Require stack:
        /Users/transverna/code/github/Howdju/react-native-share-menu/index.js
        src/App.tsx
        __tests__/App-test.tsx

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        8.542 s
Ran all test suites.

I try transformIgnorePatterns: ['node_modules|react-native-share-menu'], in jest.config.ts based loosely upon react-navigation/react-navigation#5686 (comment). But that has two problems:

1. We need to transform other things under node_modules and 2. we need to transform the syntax of react-native-share-menu.

So I end up with:

  transformIgnorePatterns: [
    // Include some extra stuff under node_modules in our babel transform
    'node_modules/(?!((@|jest-)?react-native|react-navigation|@react-navigation/.*))',
  ],

and still need to figure out Cannot find module 'react-native' from '../../react-native-share-menu/index.js'.

Other things I looked at:

• https://jestjs.io/docs/getting-started#using-typescript

Maybe this?

• https://jestjs.io/docs/getting-started#using-typescript (it seems like I should fix this by transforming correctly rather than just including in transformation.)
• https://reactnavigation.org/docs/testing/ (I'm not using reanimated yet)

[carlgieringer]

0279d67 adds CI. Next steps include:

• Increasing unit test coverage
• Adding integration tests for Web and Mobile iOS
• Adding CD to preprod and one-click deploy to prod.

[carlgieringer]

Remaining issues:

1. The deployment repeats the premerge checks. We should either remove them from the deployment script or try and detect when we are running as a Github workflow and skip them. We could detect them by adding a special env. var in our secrets: HOWDJU_RUNNING_IN_GITHUB_WORKFLOW.
2. We kick off the deploy any time the CI workflow runs. We use a condition only to run the one on the master branch, creating noise in our workflow run history. We could split up our CI to have separate actions for pushes/pull requests and pushes to master. But we would probably want to figure out how to reuse the workflow so that we didn't have redundancy.
3. We deploy everything for every push to master, but really we should only push if there were changes affecting a particular component, API, web app, mobile app, etc.