feat: genai-rag vertex search (init) #5
Conversation
| template { | ||
| template { | ||
| containers { | ||
| # Replace with ingestion job container |
There was a problem hiding this comment.
is this a comment as a TODO or is this for the sample reader?
There was a problem hiding this comment.
For the reader. While in devmode I've added TODO(glasnt) for notes for myself, rather than usage notes. Unsure how best to relay that aside from this method.
There was a problem hiding this comment.
There's a few style choices that I'm making here, for now (next commit) I'm trying Note: above when things should be customized. I also want to mark Design Considerations in a way that's obvious as well. I'll at least be consistent in this sample, but unsure how far I want to denote repo-specific styleguides around this.
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Aspects to setup ingestion from Cloud Storage bucket to processing job. |
There was a problem hiding this comment.
Don't we also need a "google_storage_notification" resource to enable notifications to be sent to the Pub/Sub topic when a new object is added or updated in the GCS bucket ?
There was a problem hiding this comment.
I thought I would be handling this through ingestion_data_source_settings, but I'd have to check what the differences are here.
There was a problem hiding this comment.
I've established that using a base of https://github.com/terraform-google-modules/terraform-docs-samples/blob/main/functions/pubsub/main.tf, adding https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_notification#example-usage to put data into the topic, and adapting the JavaScript sample to Python (on request), I have a working ingestion pipeline
| index = google_vertex_ai_index.default.id | ||
| dedicated_resources { | ||
| machine_spec { | ||
| machine_type = "n1-standard-16" |
There was a problem hiding this comment.
I recommend that we define a variable for this in variables.tf. Users might want to experiment with the machine type
to optimize index performance.
There was a problem hiding this comment.
There are a lot of elements the user might want to change in vertex.tf to optimize performance, not just the deployed index, but all the index configurations. I'll use comments rather than variables to denote this.
There was a problem hiding this comment.
In c84256c I denoted a tentative comment scheme to differentiate between settings that are Design Considerations, and settings that should be adapted by the user.
Based on the list used in project_services in main.tf
Thank you for opening a Pull Request!
Before submitting your PR, there are a few things you can do to make sure it goes smoothly: