feat: menu do usuário e ajustes em OTP/auth para UI#284
Merged
silveirado merged 24 commits intodevelopfrom Feb 5, 2026
Merged
Conversation
… streaming - Create findStream function with record-by-record processing - Extract common logic to findUtils.ts (DRY principle) - Create Transform streams for field permissions and date conversion - Add ObjectToJsonTransform for HTTP streaming - Add new endpoint /rest/stream/:document/findStream - Register streamApi in routes/index.ts - Add unit tests for Transform streams and findUtils - Add integration, E2E, and benchmark tests - Add confidence test to validate data consistency - All tests execute directly in Node (no Jest dependency) - Benchmark shows 82% memory reduction and 99% faster TTFB for 55k records TODO: Refactor and cleanup
- Extract magic numbers to streamConstants.ts (DRY)
- Replace let with const (const-pref)
- Replace forEach/for loops with functional methods (.map, .filter, .reduce)
- Extract helper functions from findUtils.ts (buildSortOptions, buildAccessConditionsForField, buildAccessConditionsMap, calculateConditionsKeys)
- Extract parseFilterFromQuery to eliminate duplication in streamApi.ts
- Create streamTestHelpers.ts with reusable test functions
- Use BluebirdPromise.map with concurrency limits in all promise operations
- Add default sort { _id: 1 } to findStream for consistent ordering
- Match find.ts behavior in findUtils.ts for query construction consistency
- Refactor test files to use helpers and functional methods
- Fix test variable references (testResults.allPassed)
All tests passing:
- Unit and integration tests: 7/7 passed
- Benchmark: 99.3% faster TTFB, 45% faster total time, 81.8% better throughput
- Confidence test: All datasets match exactly with find paginated endpoint
- Add comprehensive documentation for /rest/stream/:document/findStream endpoint - Document streaming format (newline-delimited JSON) - Include client-side processing examples (JavaScript) - Add advantages comparison with traditional find endpoint - Add usage guidelines and best practices - Update Postman collection with 3 new requests: - Find Stream (main request with all parameters) - Find Stream - Contact (simple example) - Find Stream - With Filter (complex filter example) - Documentation available in pt-BR and en - Include response examples and error handling
…ntation
- ADR-0001: HTTP Streaming para Busca de Dados
Documents decision to implement HTTP streaming endpoint
Includes performance metrics (68% memory reduction, 99.3% faster TTFB)
- ADR-0002: Extração de Lógica Comum para findUtils
Documents DRY principle application
Explains shared logic extraction between find and findStream
- ADR-0003: Node.js Transform Streams para Processamento Sequencial
Documents use of Transform streams for record-by-record processing
Explains pipeline architecture
- ADR-0004: Ordenação Padrão para Consistência
Documents default sorting decision ({ _id: 1 })
Explains consistency requirements for confidence tests
All ADRs available in pt-BR and en
Includes README files with index
- Add hasSecondaryNodes() function to check for available secondary nodes - Implement dynamic read preference selection: - Uses 'secondary' when secondaries are available (maximum isolation) - Falls back to 'secondaryPreferred' when no secondaries (no errors) - Add performance optimizations: - STREAM_BATCH_SIZE: 1000 documents per batch - STREAM_MAX_TIME_MS: 5 minutes max query time - Apply same read preference to countDocuments for consistency - Update ADR-0005 to reflect smart fallback approach - Works in all environments (dev without secondaries, prod with secondaries) See ADR-0005 for detailed rationale
- Remove KonectyResult (not used) - Remove errorReturn (not used) - Remove successReturn (not used) - Remove DataDocument (not used directly, only in streamTransforms) All imports are now used, lint passes without errors
- Add hierarchical pivot table structure with nested children - Enrich pivot config with metadata from MetaObject.Meta - Implement lookup field formatting with formatPattern - Add recursive field metadata resolution for nested lookups - Concatenate parent labels in nested fields (e.g., 'Grupo > Nome') - Calculate subtotals per hierarchy level - Calculate grand totals for all data - Update Python script to build hierarchical structure - Support Accept-Language header for multilingual labels - Update integration and unit tests for new structure Breaking changes: - Pivot API response format changed from flat array to hierarchical structure - Response now includes metadata, data (hierarchical), and grandTotals
…rmat - Update API documentation (en/pt-BR) with new hierarchical response structure - Add examples showing metadata, nested children, subtotals, and grandTotals - Document lookup formatting rules and nested field label concatenation - Add ADR-0007 documenting hierarchical pivot output format decision - Update ADR READMEs to include new ADR Breaking changes documented: - Response format changed from flat array to hierarchical structure - New metadata field with enriched field information - Nested children arrays for multi-level hierarchies - Subtotals per level and grand totals
- Update Postman collection example response to show new hierarchical structure - Include metadata, nested children, subtotals, and grandTotals in example - Reflect breaking change in response format
- Add Rust, cargo, and musl-dev for building polars from source on Alpine - Fix ENV format to use key=value syntax (removes warning) - Fix COPY paths to use absolute paths (/app instead of app) - Add python3-dev and py3-pip for Python development dependencies - Ensure konecty user has access to build tools - Note: polars will compile on first execution (takes ~2-5 minutes), then cached Alpine Linux (musl) doesn't have precompiled polars wheels, so compilation from source is required. This is handled automatically by uv when the script runs for the first time.
- Add GET /rest/data/:document/graph endpoint for SVG chart generation - Implement graphStream function orchestrating findStream + Python - Create graph_generator.py script using Polars for aggregations and pandas/matplotlib for visualization - Support 6 chart types: bar, line, pie, scatter, histogram, timeSeries - Add collectSVGFromPython function to pythonStreamBridge for SVG collection - Add GraphConfig and GraphStreamParams TypeScript types - Create unit and integration tests for graph endpoint - Add ADR-0008 documenting Polars+Pandas decision (pt-BR and en) - Update API documentation with graph endpoint examples (pt-BR and en) - Update Postman collection with graph examples using Opportunity document - Performance: Polars is 3-10x faster than Pandas for aggregations - Convert only aggregated results to Pandas (memory efficient) - Add pyarrow dependency for Polars to_pandas() conversion
- Fix wrong variable name in runFindStreamTests.ts (failed++ -> testResults.failed++) - Change BENCHMARK_ITERATION_CONCURRENCY from 3 to 1 for accurate memory measurements - Fix TypeScript linting errors (any type, empty line, type guards)
- Replace echo with printf in polars pre-build step - BusyBox ash doesn't interpret \n in echo, causing malformed input - printf correctly interprets \n as newline character - This ensures polars is properly pre-compiled during Docker build - Prevents multi-minute delay on first pivot/graph request
- Add support for hierarchical column headers in pivotStream response - Update tests to validate presence and structure of columnHeaders - Modify API documentation to reflect new columnHeaders field - Implement logic in Python script to handle and return column headers - Ensure backward compatibility with existing pivot functionality Breaking changes: - Response format now includes columnHeaders, enhancing the pivot table structure.
- Pie charts: corrigido para sempre usar categoryField + aggregation, ignorando series - Bar charts: ajustado para usar xAxis.field quando disponível (em vez de sempre categoryField) - Campos de sistema lookup: melhorada projeção para _user, _createdBy, etc. (incluem _id para população) - Normalização: graph_type normalizado para lowercase em todas as comparações - Mensagens de erro: melhoradas com listagem de campos disponíveis para debug - Logs de debug: adicionados para rastrear identificação de lookups e campos disponíveis - Resolução de campos: melhorada para campos de sistema que são lookups
- graphMetadata: gera título instrutivo quando vazio
- Inclui nome do módulo/documento
- Usa campo de agrupamento principal (categoryField ou xAxis)
- Formato: '{Módulo} por {Campo}' (pt-BR) ou '{Module} by {Field}' (en)
- Preserva títulos existentes (não sobrescreve)
- graphMetadata: adicionado comentário sobre ADR-0016 e ADR-0008 - graphStream: adicionado comentário sobre ADR-0016 e ADR-0008 - Documentação: código segue decisões arquiteturais documentadas
- Adiciona seção 'graph' no pivot Activity:pivot:Default - Configura gráfico de pizza (pie chart) por status - Agrega por contagem (count) de atividades - Título i18n: "Activities by Status" / "Atividades por Situação" - Legendas habilitadas, grade desabilitada Documentação: - docs/pt-BR/metadata-pivot-graph.md: guia completo de configuração - Estrutura JSON detalhada - Exemplos para cada tipo de gráfico - Referência de campos e agregações - Bucketing temporal (year, quarter, month, week, day, hour) - Boas práticas e validação Esta configuração permite que usuários visualizem o gráfico automaticamente ao alternar para o modo "Gráfico" na interface, sem necessidade de configuração manual. Ref: ADR-0020 (frontend)
…tegration
- Add addressSource enum field ('DNE' | 'Google') to Namespace schema
- Include addressSource in userInfo response to enable frontend address source selection
- Supports switching between DNE and Google Places Autocomplete address input
- Added optional 'includeTotal' query parameter to the findStream endpoint to return total count of results. - Implemented a new count endpoint to retrieve the total number of documents matching the specified filters. - Updated response headers to include 'X-Total-Count' when 'includeTotal' is true. - Improved error handling and logging for the new functionality.
- Revisão completa das mensagens de erro seguindo ADR-0024 - Backend retorna códigos de erro estruturados (code, message, details) - Mensagens empáticas sem terminologia técnica - Admin users bypassam verificações de permissão de export - Normalização xls→xlsx verifica permissões legadas - Erros retornam status HTTP apropriados (400/403/500) - Content-Type application/json para respostas de erro - Removidos prefixos técnicos [document] das mensagens ao usuário
- Atualizar skill codigo-limpo com p-limit (substituindo Bluebird deprecated) - Expandir error handling centralizado com express-async-errors - Adicionar padrões async modernos (for-await-of, Promise.allSettled) - Criar nova skill testing-patterns com Jest + Supertest + MongoDB Memory Server - Criar ADR-0001: Deprecação do Bluebird - Criar ADR-0002: Error Handling Centralizado - Criar ADR-0003: Estratégia de Testes com Jest - Atualizar README das skills com novas funcionalidades
- imports/menu: expansão do menu principal - api/auth/otp: melhorias no fluxo OTP - api/menu/main: ajustes no endpoint de menu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alterações para suporte à UI: