A comprehensive AI-powered medical assistant that provides intelligent health consultations through multiple input modalities including text, voice, and medical image analysis.
This project was developed for the VITB-JHU Health Hack, combining cutting-edge AI technologies to create an accessible and intelligent medical consultation platform.
BotMed/
βββ .github/
β βββ Assets/
β β βββ Contributors/
β β β βββ Rana_Talukdar.png
β β β βββ Bindupautra_Jyotibrat.png
β β β βββ Arunim_Gogoi.png
β β β βββ Ansh_Gaur.jpg
β β β βββ Akshit_Joshi.jpeg
β β β
β β βββ logo.png
β β
β βββ ISSUE_TEMPLATES/
β β βββ bug_report.md
β β βββ feature_request.md
β β
β βββ CODE_OF_CONDUCT.md
β βββ READMD.md
β
βββ Archive/
β βββ Notebooks/
β β βββ API.ipynb
β β βββ deepseek_fine_tune.ipynb
β β βββ Openai_Whisper_STT_with_NLP.ipynb
β β βββ Testing.ipynb
β β
β βββ Python Scripts/
β β βββ aud2text.py
β β βββ breath_heart_sound.py
β β βββ generate_audio.py
β β βββ heart_breath_sound.py
β β
β βββ Testing Data/
β β βββ Audio Data/
β β βββ OPUS and MP3 Audio Files/
β β β βββ test1.opus
β β β βββ test2.mp3
β β β βββ test3.opus
β β β βββ test4.opus
β β β
β β βββ WAV Audio Files/
β β βββ test1.wav
β β βββ test2.wav
β β βββ test3.wav
β β
β βββ README.md
β
βββ Backend/
β βββ Notebooks/
β β βββ Main Backend/
β β β βββ llama_8b_integrated_aud.ipynb
β β β βββ tumour_class.ipynb
β β β βββ whisper_aud_text.ipynb
β β β
β β βββ Speech To Text/
β β βββ Faster_Whisper_STT_with_NLP.ipynb
β β
β βββ README.md
β
βββ public/
β βββ favicon.ico
β
βββ src/
β βββ App.tsx
β βββ index.css
β βββ main.tsx
β βββ vite-env.d.ts
β
βββ .gitattributes
βββ .gitignore
βββ package.json
βββ package-lock.json
βββ index.html
βββ eslint.config.js
βββ postcss.config.js
βββ tailwind.config.js
βββ tsconfig.app.json
βββ tsconfig.json
βββ tsconfig.node.json
βββ LICENSE
- Text Input: Direct text-based medical queries
- Voice Input: Audio-to-text conversion using OpenAI WhisperX
- Image Analysis: Medical image classification for tumor detection
- Fine-tuned LLaMA 7B: Specialized medical knowledge base
- NLP Health Filter: Intelligent detection of health-related queries
- Medical Image Classification: ML model for tumor type identification
- Contextual Responses: Relevant and accurate medical information
- Python: Core backend development
- LLaMA 7B: Fine-tuned large language model
- OpenAI WhisperX: Audio-to-text conversion
- Custom ML Models: Image classification and NLP filtering
- TypeScript: Type-safe development
- React: Component-based UI framework
- Vite: Fast build tool and development server
- HTML5: Semantic markup
- Tailwind CSS: Utility-first styling framework
- The Models must be run on google colab or kaggle for the development setup.
- Clone the repository and navigate to the
backend
directory and then toNotebooks
directory.git clone https://github.com/Jyotibrat/BotMed.git cd "backend/Notebooks"
- Upload the Notebooks to the google colab or kaggle to run them (You can run them in your system also if you have a very good GPU).
Note: Currently the Backend is in development and the project is not fully deployed yet.
- Clone the repository and navigate to the root directory.
git clone https://github.com/Jyotibrat/BotMed.git
- Install the node modules using npm (you can also download using other package manager like
yarn
).npm install
- If the packages in needing funding.
npm fund
- If packages need to be fixed.
npm audit fix
- If the fixing of package still persists after running the above command then run this command.
npm audit fix --force
- Run the development server.
npm run dev
- Open your browser and navigate to
http://localhost:5173/
to see the application.
Note: Check the frontend deployment here
- LLaMA 7B Fine-tuning: Achieved high accuracy on medical query responses
- Image Classification: Demonstrates reliable tumor type detection
- NLP Filter: Effectively distinguishes health-related vs. non-health queries
- WhisperX: High-quality audio transcription with medical terminology support
We welcome contributions to improve the BotMed! Please read our Code of Conduct before contributing.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature
) - Commit your changes (
git commit -m 'Add amazing feature'
) - Push to the branch (
git push origin feature/amazing-feature
) - Open a Pull Request
This project is licensed under the GPL-3.0 License - see the LICENSE file for details.
Important: This medical chatbot is designed for educational and informational purposes only. It should not be used as a substitute for professional medical advice, diagnosis, or treatment. Always consult with qualified healthcare professionals for medical concerns.
This project was made possible by the contributions of these amazing individuals:
- VITB-JHU Health Hack organizers
- OpenAI for WhisperX technology
- Meta AI for LLaMA foundation model
- Healthcare professionals who provided domain expertise
For questions, suggestions, or collaboration opportunities, please reach out to us @here.