Jan Server can do Models routing #197

locnguyen1986 · 2025-09-30T07:00:36Z

Added Cloud Provider with proxy routing

OpenRouter
Gemini

Added/Updated API
Providers (org / project)

POST /v1/organization/providers — create an org-level provider (encrypted API key).
GET /v1/organization/providers — list org providers (no secret leakage).
GET /v1/organization/providers/{provider_id} — fetch one provider.
PATCH /v1/organization/providers/{provider_id} — update active, metadata, API Key
DELETE /v1/organization/providers/{provider_id} — deactivate/remove provider.
POST /v1/organization/projects/{project_id}/providers — create a project-scoped provider.
GET /v1/organization/projects/{project_id}/providers — list project providers.
GET /v1/organization/projects/{project_id}/providers/{provider_id} — fetch one project provider.
PATCH /v1/organization/projects/{project_id}/providers/{provider_id} — update project provider.
DELETE /v1/organization/projects/{project_id}/providers/{provider_id} — deactivate/remove project provider.

Model catalog & discovery

GET /v1/models/providers — merged list of providers visible to caller (Jan + org + project).
GET /v1/models — final model list with provider_id/type/vendor/name (priority: project → org → Jan).
GET /v1/conv/models — conversation-capable models with same provider metadata.-

Inference (proxy/routing)

POST /v1/chat/completions — standard chat completion (supports provider_*, stream).
POST /v1/conv/chat/completions — conversation-aware completion (conversation, store, provider override, stream).

Caching (admin)

POST /v1/admin/cache/invalidate — targeted invalidation by scope/keys.

Bug fixes

Fix streaming with /conv, use StreamMessage
Fix Cache Layer, restructure

jjchen01 · 2025-10-01T10:13:52Z

apps/jan-api-gateway/application/app/utils/httpclients/gemini/client.go

+		defer writer.Close()
+
+		reqBody := request
+		reqBody.Stream = true


Do not overwrite the function arguments.

jjchen01 · 2025-10-01T10:57:10Z

apps/jan-api-gateway/application/app/utils/httpclients/gemini/client.go

+}
+
+func (c *Client) CreateChatCompletionStream(ctx context.Context, apiKey string, request openai.ChatCompletionRequest) (io.ReadCloser, error) {
+	reader, writer := io.Pipe()


Return an error early.
re-write to something like:

req := RestyClient.R().SetBody(request) resp, err := req. SetContext(ctx). SetHeader("Authorization", fmt.Sprintf("Bearer %s", apiKey)). SetHeader("Accept-Encoding", "identity"). // <- no gzip SetHeader("Content-Type", "application/json"). SetDoNotParseResponse(true). Post(c.baseURL + "/chat/completions") if err != nil { return nil, err } if resp.IsError() { return nil, err } reader, writer := io.Pipe() go func() { defer writer.Close() defer resp.RawResponse.Body.Close() // Pipe the SSE bytes through; caller should parse "data: ..." lines. if _, err = io.Copy(writer, resp.RawResponse.Body); err != nil { writer.CloseWithError(err) return } }() return reader, nil

jjchen01 · 2025-10-01T10:58:51Z

apps/jan-api-gateway/application/app/utils/httpclients/openrouter/client.go

+	return &response, nil
+}
+
+func (c *Client) CreateChatCompletionStream(ctx context.Context, apiKey string, request openai.ChatCompletionRequest) (io.ReadCloser, error) {


Why can't OpenRouter, Jan-Inference, and Gemini have the same handler for streaming responses??

jjchen01 · 2025-10-01T11:03:53Z

apps/jan-api-gateway/application/app/interfaces/http/routes/v1/admin/cache_route.go

@@ -0,0 +1,62 @@
+package admin


Remove unnecessary endpoints and services.

the operation belongs to DevOps/SRE

jjchen01 · 2025-10-01T11:13:43Z

apps/jan-api-gateway/application/app/infrastructure/database/dbschema/model_provider.go

+func (p *ModelProvider) EtoD() *domain.ModelProvider {
+	if p == nil {
+		return nil
+	}


Unreachable code.
If it's AI-generated, please review the result yourself next time.

jjchen01 · 2025-10-01T11:35:51Z

...cation/app/infrastructure/database/repository/modelproviderrepo/model_provider_repository.go

+func (r *ModelProviderGormRepository) Update(ctx context.Context, provider *domain.ModelProvider) error {
+	model := dbschema.NewSchemaModelProvider(provider)
+	tx := r.db.GetTx(ctx)
+	return tx.WithContext(ctx).Model(&dbschema.ModelProvider{}).


query.ModelProvider.WithContext(ctx).Save(model)

jjchen01 · 2025-10-02T05:50:46Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	if err != nil {
+		reqCtx.AbortWithStatusJSON(http.StatusInternalServerError, responses.ErrorResponse{
+			Code:  "5c6c9f1f-0d6a-40a4-9fa6-823f19f8391d",
+			Error: "failed to load provider",


ErrorInstance: err

jjchen01 · 2025-10-02T05:55:28Z

apps/jan-api-gateway/application/app/domain/modelprovider/model_provider_service.go

+	encryptionSecret string
+}
+
+type CreateOrganizationProviderInput struct {


Maybe you can just use ModelProvider(domain)

jjchen01 · 2025-10-02T05:55:33Z

apps/jan-api-gateway/application/app/domain/modelprovider/model_provider_service.go

+	Active         bool
+}
+
+type UpdateOrganizationProviderInput struct {


Maybe you can just use ModelProvider(domain)

jjchen01 · 2025-10-02T05:59:05Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+		return
+	}
+
+	input := modelprovider.UpdateOrganizationProviderInput{PublicID: provider.PublicID}


How about adding a function to generate ModelProvider from the request?
It's a common pattern to convert objects to entities in DDD.

func(req)GetModelProvider() ModelProvider { ... } req.GetModelProvider()

jjchen01 · 2025-10-02T05:59:28Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+		return
+	}
+
+	provider, err := route.fetchOrganizationProvider(reqCtx.Request.Context(), orgEntity.ID, reqCtx.Param("provider_id"))


reqCtx.Param("provider_id") check bad request

jjchen01 · 2025-10-02T06:05:02Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	authService     *auth.AuthService
+	providerService *modelprovider.ModelProviderService
+	projectService  *project.ProjectService
+	cache           *cache.RedisCacheService


The route is supposed to integrate with domain services; we should move the cache to the service level.
Although named RedisCacheService, it is actually a repository.

jjchen01 · 2025-10-02T06:06:34Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	route.invalidateOrganizationModelsCache(reqCtx.Request.Context(), orgEntity.ID)
+	unlinkProviderModelsCache(reqCtx.Request.Context(), route.cache, updated.PublicID, "organization provider")


move them to providerService, maybe just after UpdateOrganizationProvider

jjchen01 · 2025-10-02T06:07:14Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	route.invalidateOrganizationModelsCache(reqCtx.Request.Context(), orgEntity.ID)
+	if provider.ProjectID != nil {
+		cacheKey := fmt.Sprintf(cache.ProjectModelsCacheKeyPattern, *provider.ProjectID)
+		if err := route.cache.Unlink(reqCtx.Request.Context(), cacheKey); err != nil {
+			logger.GetLogger().Warnf("organization provider: failed to invalidate project cache for project %d: %v", *provider.ProjectID, err)
+		}
+	}
+	unlinkProviderModelsCache(reqCtx.Request.Context(), route.cache, provider.PublicID, "organization provider")


Move them to providerService services

jjchen01 · 2025-10-02T06:07:43Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+// @Failure 401 {object} responses.ErrorResponse "Unauthorized"
+// @Failure 404 {object} responses.ErrorResponse "Provider not found"
+// @Failure 500 {object} responses.ErrorResponse "Internal server error"
+// @Router /v1/organization/projects/{proj_public_id}/providers/{provider_id} [get]


jjchen01 · 2025-10-02T06:07:58Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+		return
+	}
+
+	provider, err := route.fetchProjectProvider(reqCtx.Request.Context(), orgEntity.ID, projectEntity.ID, reqCtx.Param("provider_id"))


reqCtx.Param("provider_id")

jjchen01 · 2025-10-02T06:08:44Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	route.invalidateProjectModelsCache(reqCtx.Request.Context(), projectEntity.ID)
+	route.invalidateOrganizationModelsCache(reqCtx.Request.Context(), orgEntity.ID)
+	unlinkProviderModelsCache(reqCtx.Request.Context(), route.cache, provider.PublicID, "project provider")


The route should not be concerned with the cache.

jjchen01 · 2025-10-02T06:10:15Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+	if provider == nil || provider.OrganizationID == nil || *provider.OrganizationID != organizationID {
+		return nil, nil
+	}
+	if provider.ProjectID == nil || *provider.ProjectID != projectID {
+		return nil, nil
+	}


return error base on the context?

jjchen01 · 2025-10-02T06:10:56Z

...i-gateway/application/app/interfaces/http/routes/v1/organization/providers/provider_route.go

+func (route *ProjectProviderRoute) invalidateOrganizationModelsCache(ctx context.Context, organizationID uint) {
+	if route.cache == nil {
+		return
+	}
+	cacheKey := fmt.Sprintf(cache.OrganizationModelsCacheKeyPattern, organizationID)
+	if err := route.cache.Unlink(ctx, cacheKey); err != nil {
+		logger.GetLogger().Warnf("project provider: failed to invalidate organization cache for org %d: %v", organizationID, err)
+	}
+}
+
+func unlinkProviderModelsCache(ctx context.Context, cacheService *cache.RedisCacheService, providerID string, logPrefix string) {
+	if cacheService == nil {
+		return
+	}
+	trimmed := strings.TrimSpace(providerID)
+	if trimmed == "" {
+		return
+	}
+	cacheKey := fmt.Sprintf("%s:%s", cache.ModelsCacheKey, trimmed)
+	if err := cacheService.Unlink(ctx, cacheKey); err != nil {
+		logger.GetLogger().Warnf("%s: failed to invalidate provider models cache for provider %s: %v", logPrefix, trimmed, err)
+	}
+}


remove cache from route

locnguyen1986 added 17 commits September 30, 2025 08:42

add first version of inference provider, selection mode

07ee1e3

remove unused constants

fb2b79d

improve code to call model provides, check Gemini working

e81e6d9

fix models structures

bd08a01

add cache improvement, fix logic on active

f995647

remove unused code and improve caching

d8c5f43

add code gen

e665fd1

remove concrete InferenceProvider

b4b94bf

clean the models structure

d77f3d2

enhance filtering

cf4aa69

fix constants

6eaaf20

update swagger doc

26f16f5

update cache solutions

338b8b6

add missing APIs in provider router

9dce62f

add invalidate cache endpoint

0f86377

improve streaming with conv

2d82f47

fix gemini

dc8e574

locnguyen1986 requested a review from jjchen01 September 30, 2025 15:16

jjchen01 reviewed Oct 1, 2025

View reviewed changes

jjchen01 reviewed Oct 2, 2025

View reviewed changes

locnguyen1986 closed this Oct 8, 2025

		route.invalidateOrganizationModelsCache(reqCtx.Request.Context(), orgEntity.ID)
		unlinkProviderModelsCache(reqCtx.Request.Context(), route.cache, updated.PublicID, "organization provider")

Jan Server can do Models routing #197

Jan Server can do Models routing #197

Uh oh!

Conversation

locnguyen1986 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

locnguyen1986 commented Sep 30, 2025 •

edited

Loading