-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Update what is being audit logged #11833
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
FelixMalfait
wants to merge
19
commits into
main
Choose a base branch
from
change-event-logging
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
69a9d7d
Update what is being audit logged
FelixMalfait ea88a13
Disable monitoring
FelixMalfait a1cf25c
Review
FelixMalfait fded81e
Begin refacto
FelixMalfait b342991
Refactoring
FelixMalfait 8c872c1
Fix casing
FelixMalfait ef0c9f1
Lint
FelixMalfait 9829f16
Folder casing issue
FelixMalfait 4e8a2b8
More renaming
FelixMalfait 1cdb7ff
Renaming
FelixMalfait f5acf00
Lint
FelixMalfait 069b3a2
Fix test
FelixMalfait e052929
Remove test
FelixMalfait c06d907
More renaming
FelixMalfait 85577f4
Update packages/twenty-server/src/database/clickHouse/migrations/003-…
FelixMalfait f28b0d8
Update packages/twenty-server/src/database/clickHouse/migrations/003-…
FelixMalfait 1dec4c8
Update packages/twenty-server/src/engine/core-modules/audit/README.md
FelixMalfait 756f818
Test
FelixMalfait 3f8b62f
More renaming
FelixMalfait File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
12 changes: 12 additions & 0 deletions
12
packages/twenty-server/src/database/clickHouse/clickHouse.module.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
import { Module } from '@nestjs/common'; | ||
|
||
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module'; | ||
|
||
import { ClickHouseService } from './clickHouse.service'; | ||
|
||
@Module({ | ||
imports: [TwentyConfigModule], | ||
providers: [ClickHouseService], | ||
exports: [ClickHouseService], | ||
}) | ||
export class ClickHouseModule {} |
264 changes: 264 additions & 0 deletions
264
packages/twenty-server/src/database/clickHouse/clickHouse.service.spec.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,264 @@ | ||
import { Test, TestingModule } from '@nestjs/testing'; | ||
|
||
import { ClickHouseClient } from '@clickhouse/client'; | ||
|
||
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service'; | ||
|
||
import { ClickHouseService } from './clickHouse.service'; | ||
|
||
// Mock the createClient function from @clickhouse/client | ||
jest.mock('@clickhouse/client', () => ({ | ||
createClient: jest.fn().mockReturnValue({ | ||
insert: jest.fn().mockResolvedValue({}), | ||
query: jest.fn().mockResolvedValue({ | ||
json: jest.fn().mockResolvedValue([{ test: 'data' }]), | ||
}), | ||
ping: jest.fn().mockResolvedValue({ success: true }), | ||
close: jest.fn().mockResolvedValue({}), | ||
exec: jest.fn().mockResolvedValue({}), | ||
}), | ||
})); | ||
|
||
describe('ClickHouseService', () => { | ||
let service: ClickHouseService; | ||
let twentyConfigService: TwentyConfigService; | ||
let mockClickHouseClient: jest.Mocked<ClickHouseClient>; | ||
|
||
beforeEach(async () => { | ||
jest.clearAllMocks(); | ||
|
||
mockClickHouseClient = { | ||
insert: jest.fn().mockResolvedValue({}), | ||
query: jest.fn().mockResolvedValue({ | ||
json: jest.fn().mockResolvedValue([{ test: 'data' }]), | ||
}), | ||
ping: jest.fn().mockResolvedValue({ success: true }), | ||
close: jest.fn().mockResolvedValue({}), | ||
exec: jest.fn().mockResolvedValue({}), | ||
} as unknown as jest.Mocked<ClickHouseClient>; | ||
|
||
const module: TestingModule = await Test.createTestingModule({ | ||
providers: [ | ||
ClickHouseService, | ||
{ | ||
provide: TwentyConfigService, | ||
useValue: { | ||
get: jest.fn((key) => { | ||
if (key === 'CLICKHOUSE_URL') return 'http://localhost:8123'; | ||
|
||
return undefined; | ||
}), | ||
}, | ||
}, | ||
], | ||
}).compile(); | ||
|
||
service = module.get<ClickHouseService>(ClickHouseService); | ||
twentyConfigService = module.get<TwentyConfigService>(TwentyConfigService); | ||
|
||
// Set the mock client | ||
(service as any).mainClient = mockClickHouseClient; | ||
}); | ||
|
||
it('should be defined', () => { | ||
expect(service).toBeDefined(); | ||
}); | ||
|
||
describe('constructor', () => { | ||
it('should not initialize clickhouse client when clickhouse is disabled', async () => { | ||
jest.spyOn(twentyConfigService, 'get').mockImplementation((key) => { | ||
if (key === 'CLICKHOUSE_URL') return ''; | ||
|
||
return undefined; | ||
}); | ||
|
||
const newModule: TestingModule = await Test.createTestingModule({ | ||
providers: [ | ||
ClickHouseService, | ||
{ | ||
provide: TwentyConfigService, | ||
useValue: twentyConfigService, | ||
}, | ||
], | ||
}).compile(); | ||
|
||
const newService = newModule.get<ClickHouseService>(ClickHouseService); | ||
|
||
expect((newService as any).mainClient).toBeUndefined(); | ||
}); | ||
}); | ||
|
||
describe('insert', () => { | ||
it('should insert data into clickhouse and return success', async () => { | ||
const testData = [{ id: 1, name: 'test' }]; | ||
const result = await service.insert('test_table', testData); | ||
|
||
expect(result).toEqual({ success: true }); | ||
expect(mockClickHouseClient.insert).toHaveBeenCalledWith({ | ||
table: 'test_table', | ||
values: testData, | ||
format: 'JSONEachRow', | ||
}); | ||
}); | ||
|
||
it('should return failure when clickhouse client is not defined', async () => { | ||
(service as any).mainClient = undefined; | ||
|
||
const testData = [{ id: 1, name: 'test' }]; | ||
const result = await service.insert('test_table', testData); | ||
|
||
expect(result).toEqual({ success: false }); | ||
}); | ||
|
||
it('should handle errors and return failure', async () => { | ||
const testError = new Error('Test error'); | ||
|
||
mockClickHouseClient.insert.mockRejectedValueOnce(testError); | ||
|
||
const testData = [{ id: 1, name: 'test' }]; | ||
const result = await service.insert('test_table', testData); | ||
|
||
expect(result).toEqual({ success: false }); | ||
// Since the service uses logger.error instead of exceptionHandlerService.captureExceptions, | ||
// we don't need to assert on exceptionHandlerService | ||
}); | ||
}); | ||
|
||
describe('select', () => { | ||
it('should execute a query and return results', async () => { | ||
const query = 'SELECT * FROM test_table WHERE id = {id:Int32}'; | ||
const params = { id: 1 }; | ||
|
||
mockClickHouseClient.query.mockResolvedValueOnce({ | ||
json: jest.fn().mockResolvedValueOnce([{ id: 1, name: 'test' }]), | ||
} as any); | ||
|
||
const result = await service.select(query, params); | ||
|
||
expect(result).toEqual([{ id: 1, name: 'test' }]); | ||
expect(mockClickHouseClient.query).toHaveBeenCalledWith({ | ||
query, | ||
format: 'JSONEachRow', | ||
query_params: params, | ||
}); | ||
}); | ||
|
||
it('should return empty array when clickhouse client is not defined', async () => { | ||
(service as any).mainClient = undefined; | ||
|
||
const query = 'SELECT * FROM test_table'; | ||
const result = await service.select(query); | ||
|
||
expect(result).toEqual([]); | ||
}); | ||
|
||
it('should handle errors and return empty array', async () => { | ||
const testError = new Error('Test error'); | ||
|
||
mockClickHouseClient.query.mockRejectedValueOnce(testError); | ||
|
||
const query = 'SELECT * FROM test_table'; | ||
const result = await service.select(query); | ||
|
||
expect(result).toEqual([]); | ||
// Since the service uses logger.error instead of exceptionHandlerService.captureExceptions, | ||
// we don't need to assert on exceptionHandlerService | ||
}); | ||
}); | ||
|
||
describe('createDatabase', () => { | ||
it('should create a database and return true', async () => { | ||
const result = await service.createDatabase('test_db'); | ||
|
||
expect(result).toBe(true); | ||
expect(mockClickHouseClient.exec).toHaveBeenCalledWith({ | ||
query: 'CREATE DATABASE IF NOT EXISTS test_db', | ||
}); | ||
}); | ||
|
||
it('should return false when clickhouse client is not defined', async () => { | ||
(service as any).mainClient = undefined; | ||
|
||
const result = await service.createDatabase('test_db'); | ||
|
||
expect(result).toBe(false); | ||
}); | ||
}); | ||
|
||
describe('dropDatabase', () => { | ||
it('should drop a database and return true', async () => { | ||
const result = await service.dropDatabase('test_db'); | ||
|
||
expect(result).toBe(true); | ||
expect(mockClickHouseClient.exec).toHaveBeenCalledWith({ | ||
query: 'DROP DATABASE IF EXISTS test_db', | ||
}); | ||
}); | ||
|
||
it('should return false when clickhouse client is not defined', async () => { | ||
(service as any).mainClient = undefined; | ||
|
||
const result = await service.dropDatabase('test_db'); | ||
|
||
expect(result).toBe(false); | ||
}); | ||
}); | ||
|
||
describe('connectToClient', () => { | ||
it('should connect to a client and return it', async () => { | ||
jest | ||
.spyOn(service, 'connectToClient') | ||
.mockResolvedValueOnce(mockClickHouseClient); | ||
Comment on lines
+209
to
+211
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. logic: Mocking connectToClient while testing connectToClient creates a circular dependency and may mask real issues. Consider testing the actual implementation instead. |
||
|
||
const result = await service.connectToClient('test-client'); | ||
|
||
expect(result).toBe(mockClickHouseClient); | ||
}); | ||
|
||
it('should reuse an existing client if available', async () => { | ||
// Set up a client in the map | ||
(service as any).clients.set('test-client', mockClickHouseClient); | ||
|
||
const result = await service.connectToClient('test-client'); | ||
|
||
expect(result).toBe(mockClickHouseClient); | ||
}); | ||
}); | ||
|
||
describe('disconnectFromClient', () => { | ||
it('should disconnect from a client', async () => { | ||
// Set up a client in the map | ||
(service as any).clients.set('test-client', mockClickHouseClient); | ||
|
||
await service.disconnectFromClient('test-client'); | ||
|
||
expect(mockClickHouseClient.close).toHaveBeenCalled(); | ||
expect((service as any).clients.has('test-client')).toBe(false); | ||
}); | ||
|
||
it('should do nothing if client does not exist', async () => { | ||
await service.disconnectFromClient('non-existent-client'); | ||
|
||
expect(mockClickHouseClient.close).not.toHaveBeenCalled(); | ||
}); | ||
}); | ||
|
||
describe('lifecycle hooks', () => { | ||
it('should ping server on module init', async () => { | ||
await service.onModuleInit(); | ||
|
||
expect(mockClickHouseClient.ping).toHaveBeenCalled(); | ||
}); | ||
|
||
it('should close all clients on module destroy', async () => { | ||
// Set up a couple of clients | ||
(service as any).clients.set('client1', mockClickHouseClient); | ||
(service as any).clients.set('client2', mockClickHouseClient); | ||
|
||
await service.onModuleDestroy(); | ||
|
||
// One for mainClient, and two for the clients in the map | ||
expect(mockClickHouseClient.close).toHaveBeenCalledTimes(3); | ||
}); | ||
}); | ||
}); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
style: Type assertion to 'any' could hide type mismatches. Consider creating a proper mock type that matches ClickHouseClient's QueryResult interface.