Enhance job summary metrics and reporting #210

austenstone · 2025-06-02T19:44:46Z

This pull request introduces updates to workflows, documentation, testing, and configuration files to improve functionality, maintainability, and usability. Key changes include adjustments to GitHub Actions workflows, enhancements to testing files, updates to the README.md examples, and a major restructuring of the ESLint configuration.

Workflow Updates:

.github/workflows/ci.yml: Added continue-on-error: true to the linting step to prevent workflow failures due to linting errors.
.github/workflows/usage.yml: Added pull_request trigger to monitor changes in the main branch while ignoring .md files.
.github/workflows/codeql.yml: Removed the CodeQL workflow, including its configuration for analyzing JavaScript and TypeScript code.

Dependency and Version Updates:

README.md: Updated the GitHub Action austenstone/copilot-usage from version v5.0 to v5.1 across multiple usage examples. [1] [2] [3] [4] [5]
package.json: Incremented the version of the project from 4.2.0 to 5.1.0.

Test Enhancements:

__tests__/main.test.ts: Added new test cases for createJobSummaryCopilotDetails and createJobSummarySeatAssignments functions, along with mock data for Copilot details and seat assignments. [1] [2]
__tests__/mock/sample-copilot-details.json: Introduced mock data for Copilot seat breakdown and policies.

Copilot Usage Reporting Improvements:

__tests__/mock/sample-output.md: Enhanced the reporting format for Copilot usage metrics, including detailed tables, charts, and breakdowns for daily activity, acceptance rates, and language/editor usage.
__tests__/mock/sample-copilot-details-output.md: Added a new output file summarizing Copilot seat information, policies, and breakdowns.

Codebase Simplification:

eslint.config.mjs: Refactored the ESLint configuration to simplify the setup using defineConfig and removed redundant imports and compatibility layers.

- Updated README.md to include example for team usage. - Removed console log from createJobSummaryUsage test. - Refactored job-summary.ts: - Adjusted ChartConfig interface to make title optional. - Modified dateFormat function to exclude year from formatting. - Improved createMermaidChart function to handle optional title. - Added new groupModelMetrics function for model-specific metrics. - Enhanced createJobSummaryUsage to include daily totals and new metrics for dotcom chat and PR summaries. - Updated chart generation for daily engaged users, suggestions vs. acceptances, and acceptance rates by language. - Improved pie chart generation for language and editor usage based on engaged users.

… groupModelMetrics function

…type to 'unknown' for better type safety

- Bump @actions/artifact from 2.2.1 to 2.3.2 - Bump @actions/github from 6.0.0 to 6.0.1 - Bump @octokit/action from 7.0.0 to 8.0.2 - Bump @octokit/rest from 21.1.1 to 22.0.0 - Update json-2-csv from 5.5.8 to 5.5.9 - Update jstoxml from 5.0.2 to 7.0.1 - Update octokit from 4.1.1 to 5.0.3 - Bump @types/node from 22.13.4 to 22.15.29 - Add @eslint/js as a new devDependency - Update dotenv from 16.4.7 to 16.5.0 - Update eslint from 9.20.1 to 9.28.0 - Add globals as a new devDependency - Update ts-jest from 29.2.5 to 29.3.4 - Update typescript from 5.7.3 to 5.8.3 - Update typescript-eslint from 8.24.0 to 8.33.1 - Update vitest from 3.0.5 to 3.2.0

- Added summary reporting for Copilot seat information using the new summary API from @actions/core. - Removed deprecated function createJobSummarySeatInfo and replaced its usage with summary.addTable for better formatting. - Updated imports to include the summary module from @actions/core. - Cleaned up unused imports related to job summary functions.

…ents - Added console logging for organization copilot details and seat assignments in `run.ts`. - Introduced a file output for organization copilot details as `copilot-organization-details.json`. - Enhanced visibility into the organization seat assignments by logging the sorted seat data.

…olicies table

…matting - Implemented createMermaidChart function to generate mermaid chart configurations for pie and xy charts. - Added createPieChart function to create pie charts from provided data. - Developed createXYChart function to generate xy charts with customizable series and axes. - Introduced generateLegend function to create a legend for the charts with color coding. - Added dateFormat utility function for consistent date formatting based on timezone. - Defined default chart configurations for standard height and daily categories.

…and tests feat: add mermaid chart for model usage by engaged users in sample output refactor: enhance job summary usage function for better metrics aggregation

Copilot

Pull Request Overview

This PR enhances Copilot usage reporting by refactoring utilities, updating workflows, improving tests, and simplifying configuration.

Added a reusable date formatting helper and a new mermaid chart generator module.
Replaced deprecated summary code with updated createJobSummary* functions and integrated advanced reporting in run.ts.
Bumped project and dependency versions, restructured the ESLint config, and improved GitHub Actions workflows.

Reviewed Changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/utility.ts	Introduced `dateFormat` helper with timezone support
src/run.ts	Switched to new summary functions, added output links and warnings
src/mermaid.ts	New module for generating Mermaid charts (pie & XY)
src/deprecated-job-summary.ts	Removed old summary implementation
package.json	Updated project version and bumped dependency versions
eslint.config.mjs	Simplified ESLint setup using `defineConfig` and removed legacy imports
.github/workflows/ci.yml	Enabled `continue-on-error` on lint step
tests/main.test.ts	Added tests for Copilot details and seats, updated imports

Comments suppressed due to low confidence (2)

tests/main.test.ts:28

The test references sample-copilot-seats.json, but this fixture file isn't present. Add the mock JSON or correct the path to the existing fixture.

const sampleCopilotSeats = readFileSync('./__tests__/mock/sample-copilot-seats.json', 'utf-8');

.github/workflows/ci.yml:22

The continue-on-error key must be indented under the - run: npm run lint step (as a property of that step) for it to take effect.

continue-on-error: true

Copilot · 2025-06-03T00:56:18Z

src/utility.ts

+  options.timeZone = process.env.TZ || 'UTC';
+  return new Date(date).toLocaleDateString('en-US', options);


Avoid mutating the options parameter directly. Clone it (e.g., const opts = { ...options, timeZone: process.env.TZ || 'UTC' };) before passing to toLocaleDateString to prevent side effects.

Suggested change

options.timeZone = process.env.TZ || 'UTC';

return new Date(date).toLocaleDateString('en-US', options);

const opts = { ...options, timeZone: process.env.TZ || 'UTC' };

return new Date(date).toLocaleDateString('en-US', opts);

Copilot · 2025-06-03T00:56:19Z

eslint.config.mjs

-            ...jest.environments.globals.globals,
-        },
+import globals from "globals";
+import tseslint from "typescript-eslint";


The import path typescript-eslint is incorrect. You should import the plugin from @typescript-eslint/eslint-plugin (e.g., import tsplugin from '@typescript-eslint/eslint-plugin';).

Suggested change

import tseslint from "typescript-eslint";

import tseslint from "@typescript-eslint/eslint-plugin";

austenstone added 5 commits June 2, 2025 15:43

ci: Allow linting to continue on error for improved CI resilience

b71339c

refactor: Update sumNestedValue traversal type and comment out unused…

bbe6856

… groupModelMetrics function

Refactor sumNestedValue function: Update traverse function parameter …

0c084b9

…type to 'unknown' for better type safety

chore: Remove deprecated CodeQL workflow file

a3ad28d

austenstone closed this Jun 2, 2025

austenstone reopened this Jun 2, 2025

austenstone added 12 commits June 2, 2025 16:02

Refactor code structure for improved readability and maintainability

fbdccb0

fix: add pull_request trigger to usage workflow for better CI coverage

47d8aae

fix: update seat management setting display in job summary for clarity

4b36ca5

Refactor code structure for improved readability and maintainability

e3d39cc

Update run.js.map with new source mappings and optimizations

3809ec7

Refactor code structure for improved readability and maintainability

7f5da6d

Enhance organization details reporting with additional headings and p…

82b2f2d

…olicies table

chore: update version to 5.1.0 and change usage references in README …

12e97ab

…and tests feat: add mermaid chart for model usage by engaged users in sample output refactor: enhance job summary usage function for better metrics aggregation

austenstone requested a review from Copilot June 3, 2025 00:53

Copilot AI reviewed Jun 3, 2025

View reviewed changes

austenstone merged commit 6c0eaf7 into main Jun 3, 2025
6 checks passed

austenstone deleted the rewrite branch June 3, 2025 00:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Enhance job summary metrics and reporting #210

Enhance job summary metrics and reporting #210

Uh oh!

austenstone commented Jun 2, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 3, 2025

Uh oh!

Copilot AI Jun 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		options.timeZone = process.env.TZ \|\| 'UTC';
		return new Date(date).toLocaleDateString('en-US', options);

	import tseslint from "typescript-eslint";
	import tseslint from "@typescript-eslint/eslint-plugin";

Uh oh!

Enhance job summary metrics and reporting #210

Enhance job summary metrics and reporting #210

Uh oh!

Conversation

austenstone commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Workflow Updates:

Dependency and Version Updates:

Test Enhancements:

Copilot Usage Reporting Improvements:

Codebase Simplification:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

austenstone commented Jun 2, 2025 •

edited

Loading