alternative ways to optimize vector filtering

[saihaj]

wondering if we rewrite this as CTE that might speed things up? need to check some queries to confirm.

DOGE-AI/agent/src/twitter/knowledge-base.ts

Lines 104 to 125 in ba89166

	const embeddingsQuery = await db
	.select({
	text: billVector.text,
	documentId: billVector.document,
	title: document.title,
	source: document.source,
	distance: sql`vector_distance_cos(${billVector.vector}, vector32(${termEmbeddingString}))`,
	})
	.from(billVector)
	.where(
	and(
	sql`vector_distance_cos(${billVector.vector}, vector32(${termEmbeddingString})) < ${VECTOR_SEARCH_MATCH_THRESHOLD}`,
	isNotNull(billVector.document),
	eq(billVector.source, WEB_SOURCE),
	),
	)
	.orderBy(
	sql`vector_distance_cos(${billVector.vector}, vector32(${termEmbeddingString})) ASC`,
	)
	.leftJoin(document, eq(document.id, billVector.document))
	.limit(LIMIT);