Releases: SSC-ICT-Innovatie/nl-kat-coordination
v1.5.2: documentation update: introduction into OpenKAT (#103)
- Fix missing file in rocky Debian package
- Fix timezone issue in scheduler
v1.5.1
v1.5.0
https://docs.openkat.nl/release_notes/1.5.html
OpenKAT 1.5: multi organization support
The main feature of the 1.5 release of OpenKAT is the ability to scan multiple organizations from one OpenKAT instance and manage the data and users for each of them. It comes with an API to automatically deploy organizations and users from an external source.
This gives you the ability to manage OpenKAT for a large group of organizations while maintaining the separation of data and users for each of them. It also allows you to supply credentials to users within those organizations and to give them access to their own data, make reports etc.
In addition to this several other features have been added that support this use case, such as the ability to filter and manage and add objects in bulk and add scanning and inheritance of scan levels from a higher level. They were requested by some of our large scale partners and we will continue working on such features in the forthcoming releases.
Features:
- Most importantly, this is the first version of KAT that offers fully implemented multi-organization support. New organizations can be added in the Rocky admin interface (/admin), after which databases in xtdb and KAT-alogus will be automatically added. Clients can be added for each organization that cannot view data of other organizations
- Certificates can now be created without having a relation to a website
- IPAddresses now inherit a scan level from a netblock if that netblock is known
- NMAP can now scan ip ranges
- Rocky now offers bulk actions for the deletion of OOIs and giving clearance levels to OOIs
- Rocky now offers filtering options based on clearance levels and clearance level types
IMPORTANT
This feature includes a new version of XTDB. Therefore, the env setting XTDB_TYPE has to be set to xtdb-multinode. Also, all data in xtdb will be lost and cannot be moved to this version. If you use a locally installed version, you can change to the xtdb version in this repo: https://github.com/dekkers/xtdb-http-multinode.
Full Changelog
Coordination
What's Changed
- Dynamically checkout the latest release tag with
clone-stableby @Darwinkel in #96 - Give CREATEDB rights to rocky db user by @dekkers in #98
- Update Sphinx docs by @Darwinkel in #97
- Set KAT_VERSION in .env-dist to v1.4.0 by @dekkers in #104
- Enable auto-reload of code for all services by @dekkers in #99
- Passing over xtdb work from Jeroen by @Rieven in #92
- documentation update: introduction into OpenKAT by @ring-ring-ring in #103
New Contributors
Full Changelog: v1.4.0rc1...v1.5.0rc2
Octopoes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-octopoes#31
- Feature/scan level filtering by @Lisser in minvws/nl-kat-octopoes#21
- Update
Certificatemodel by @ammar92 in minvws/nl-kat-octopoes#25 - Feature/scan profile filtering by @Lisser in minvws/nl-kat-octopoes#22
- Add Python 3.11 to the unit test matrix by @Darwinkel in minvws/nl-kat-octopoes#29
- [OpenKAT v1.5] Support multinode XTDB by @Rieven in minvws/nl-kat-octopoes#20
- Add image meta data model v1 by @Donnype in minvws/nl-kat-octopoes#38
- implement by @Lisser in minvws/nl-kat-octopoes#59
- Updated dependabot config by @ammar92 in minvws/nl-kat-octopoes#60
New Contributors
- @Lisser made their first contribution in minvws/nl-kat-octopoes#21
- @ammar92 made their first contribution in minvws/nl-kat-octopoes#25
- @Darwinkel made their first contribution in minvws/nl-kat-octopoes#29
- @Rieven made their first contribution in minvws/nl-kat-octopoes#20
- @Donnype made their first contribution in minvws/nl-kat-octopoes#38
Full Changelog: minvws/nl-kat-octopoes@v1.4.0rc1...v1.5.0rc2
Bytes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-bytes#28
- Test multiple Python versions through Docker and Makefile build arg by @Darwinkel in minvws/nl-kat-bytes#25
- Bump setuptools from 62.3.2 to 65.6.3 by @dependabot in minvws/nl-kat-bytes#27
- Optional input_ooi by @Donnype in minvws/nl-kat-bytes#20
New Contributors
- @Darwinkel made their first contribution in minvws/nl-kat-bytes#25
- @Donnype made their first contribution in minvws/nl-kat-bytes#20
Full Changelog: minvws/nl-kat-bytes@v1.4.0rc1...v1.5.0rc2
Boefjes
What's Changed
- Censys normalizer updates by @ammar92 in minvws/nl-kat-boefjes#26
- [OpenKAT v1.5] make organisation field longer by @Rieven in minvws/nl-kat-boefjes#37
- Release 1.4 by @dekkers in minvws/nl-kat-boefjes#52
- Fix plugins related to
Certificateby @ammar92 in minvws/nl-kat-boefjes#41 - [Nmap] Introduce Nmap ports boefje by @zcrt in minvws/nl-kat-boefjes#40
- [Nmap] Add settings for top-ports and protocol by @zcrt in minvws/nl-kat-boefjes#43
- [bug] RE: LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#56
- Change workflow file to test multiple Python versions by @Darwinkel in minvws/nl-kat-boefjes#44
- [Nmap] [NetBlock] ✨ introduce Nmap IP range scanning by @zcrt in minvws/nl-kat-boefjes#58
- Fix nmap schema and add test showing integers are not yet supported by @Donnype in minvws/nl-kat-boefjes#60
- Also fix nmap_ip_range integer by @Donnype in minvws/nl-kat-boefjes#68
- implement connection between ipaddresses and netblocks by @Lisser in minvws/nl-kat-boefjes#69
- [OpenKAT v1.5] Do not create _dev organisation in seed by @dekkers in minvws/nl-kat-boefjes#46
- Updated dependabot config by @ammar92 in minvws/nl-kat-boefjes#70
New Contributors
- @Rieven made their first contribution in minvws/nl-kat-boefjes#37
- @Darwinkel made their first contribution in minvws/nl-kat-boefjes#44
Full Changelog: minvws/nl-kat-boefjes@v1.4.0...v1.5.0rc2
Rocky
What's Changed
- Release 1.4.0 by @noamblitz in minvws/nl-kat-rocky#75
- add missing dmarc, dkim and spf findings by @noamblitz in minvws/nl-kat-rocky#35
- Fix ignored flake8 errors and change tests to pytest by @dekkers in minvws/nl-kat-rocky#87
- fix boefje settings forms for integer fields by @noamblitz in minvws/nl-kat-rocky#91
- untested but adding cached network reference support to csv importer. by @underdarknl in minvws/nl-kat-rocky#95
- fix flake8 error on katalogus forms by @underdarknl in minvws/nl-kat-rocky#99
- Release 1.4 by @dekkers in minvws/nl-kat-rocky#102
- Run pytest in CI by @dekkers in minvws/nl-kat-rocky#113
- Bump json5 from 2.2.1 to 2.2.3 by @dependabot in minvws/nl-kat-rocky#103
- feature(scan level filter): implement by @Lisser in minvws/nl-kat-rocky#82
- CSV import introspection by @ammar92 in minvws/nl-kat-rocky#98
- Change Organization model and add API by @dekkers in minvws/nl-kat-rocky#109
- Feature/scan profile filter by @Lisser in minvws/nl-kat-rocky#83
- [Feature] [#C86] Add certificate expiry check by @zcrt in minvws/nl-kat-rocky#88
- Updated file structure for css files within rocky. by @HeleenSG in minvws/nl-kat-rocky#122
- fix(ooi_report): take description from finding_type if None in Finding by @Lisser in minvws/nl-kat-rocky#118
- Update landing_page.html by @ring-ring-ring in minvws/nl-kat-rocky#124
- Added tag styles and colors. Updated to manon v1.1.5 by @HeleenSG in minvws/nl-kat-rocky#133
- Add support for XTDB multinode by @dekkers in minvws/nl-kat-rocky#123
- Add task list params by @jpbruinsslot in minvws/nl-kat-rocky#89
- Fix scan history template artifact by @jpbruinsslot in minvws/nl-kat-rocky#135
- [OpenKAT v1.5] Fix boefje detail checkboxes function at scan oois by @Rieven in minvws/nl-kat-rocky#111
- Feature/multi select by @Donnype in minvws/nl-kat-rocky#86
- Fix/scan levels by @Donnype in minvws/nl-kat-rocky#154
- [OpenKAT v1.5] Multi XTDB and organizations management through organization code as slug by @Rieven in minvws/nl-kat-rocky#115
- Ignore minor updates by @ammar92 in https://github.com/minvws/nl-kat-rock...
v1.5.0rc3
Full Changelog: v1.5.0rc2...v1.5.0rc3
v1.5.0rc2
OpenKAT 1.5: multi organization support
The main feature of the 1.5 release of OpenKAT is the ability to scan multiple organizations from one OpenKAT instance and manage the data and users for each of them. It comes with an API to automatically deploy organizations and users from an external source.
This gives you the ability to manage OpenKAT for a large group of organizations while maintaining the separation of data and users for each of them. It also allows you to supply credentials to users within those organizations and to give them access to their own data, make reports etc.
In addition to this several other features have been added that support this use case, such as the ability to filter and manage and add objects in bulk and add scanning and inheritance of scan levels from a higher level. They were requested by some of our large scale partners and we will continue working on such features in the forthcoming releases.
Features:
- Most importantly, this is the first version of KAT that offers fully implemented multi-organization support. New organizations can be added in the Rocky admin interface (/admin), after which databases in xtdb and KAT-alogus will be automatically added. Clients can be added for each organization that cannot view data of other organizations
- Certificates can now be created without having a relation to a website
- IPAddresses now inherit a scan level from a netblock if that netblock is known
- NMAP can now scan ip ranges
- Rocky now offers bulk actions for the deletion of OOIs and giving clearance levels to OOIs
- Rocky now offers filtering options based on clearance levels and clearance level types
IMPORTANT
This feature includes a new version of XTDB. Therefore, the env setting XTDB_TYPE has to be set to xtdb-multinode. Also, all data in xtdb will be lost and cannot be moved to this version. If you use a locally installed version, you can change to the xtdb version in this repo: https://github.com/dekkers/xtdb-http-multinode.
Full Changelog
Coordination
What's Changed
- Dynamically checkout the latest release tag with
clone-stableby @Darwinkel in #96 - Give CREATEDB rights to rocky db user by @dekkers in #98
- Update Sphinx docs by @Darwinkel in #97
- Set KAT_VERSION in .env-dist to v1.4.0 by @dekkers in #104
- Enable auto-reload of code for all services by @dekkers in #99
- Passing over xtdb work from Jeroen by @Rieven in #92
- documentation update: introduction into OpenKAT by @ring-ring-ring in #103
New Contributors
Full Changelog: v1.4.0rc1...v1.5.0rc2
Octopoes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-octopoes#31
- Feature/scan level filtering by @Lisser in minvws/nl-kat-octopoes#21
- Update
Certificatemodel by @ammar92 in minvws/nl-kat-octopoes#25 - Feature/scan profile filtering by @Lisser in minvws/nl-kat-octopoes#22
- Add Python 3.11 to the unit test matrix by @Darwinkel in minvws/nl-kat-octopoes#29
- [OpenKAT v1.5] Support multinode XTDB by @Rieven in minvws/nl-kat-octopoes#20
- Add image meta data model v1 by @Donnype in minvws/nl-kat-octopoes#38
- implement by @Lisser in minvws/nl-kat-octopoes#59
- Updated dependabot config by @ammar92 in minvws/nl-kat-octopoes#60
New Contributors
- @Lisser made their first contribution in minvws/nl-kat-octopoes#21
- @ammar92 made their first contribution in minvws/nl-kat-octopoes#25
- @Darwinkel made their first contribution in minvws/nl-kat-octopoes#29
- @Rieven made their first contribution in minvws/nl-kat-octopoes#20
- @Donnype made their first contribution in minvws/nl-kat-octopoes#38
Full Changelog: minvws/nl-kat-octopoes@v1.4.0rc1...v1.5.0rc2
Bytes
What's Changed
- Release 1.4 by @dekkers in minvws/nl-kat-bytes#28
- Test multiple Python versions through Docker and Makefile build arg by @Darwinkel in minvws/nl-kat-bytes#25
- Bump setuptools from 62.3.2 to 65.6.3 by @dependabot in minvws/nl-kat-bytes#27
- Optional input_ooi by @Donnype in minvws/nl-kat-bytes#20
New Contributors
- @Darwinkel made their first contribution in minvws/nl-kat-bytes#25
- @Donnype made their first contribution in minvws/nl-kat-bytes#20
Full Changelog: minvws/nl-kat-bytes@v1.4.0rc1...v1.5.0rc2
Boefjes
What's Changed
- Censys normalizer updates by @ammar92 in minvws/nl-kat-boefjes#26
- [OpenKAT v1.5] make organisation field longer by @Rieven in minvws/nl-kat-boefjes#37
- Release 1.4 by @dekkers in minvws/nl-kat-boefjes#52
- Fix plugins related to
Certificateby @ammar92 in minvws/nl-kat-boefjes#41 - [Nmap] Introduce Nmap ports boefje by @zcrt in minvws/nl-kat-boefjes#40
- [Nmap] Add settings for top-ports and protocol by @zcrt in minvws/nl-kat-boefjes#43
- [bug] RE: LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#56
- Change workflow file to test multiple Python versions by @Darwinkel in minvws/nl-kat-boefjes#44
- [Nmap] [NetBlock] ✨ introduce Nmap IP range scanning by @zcrt in minvws/nl-kat-boefjes#58
- Fix nmap schema and add test showing integers are not yet supported by @Donnype in minvws/nl-kat-boefjes#60
- Also fix nmap_ip_range integer by @Donnype in minvws/nl-kat-boefjes#68
- implement connection between ipaddresses and netblocks by @Lisser in minvws/nl-kat-boefjes#69
- [OpenKAT v1.5] Do not create _dev organisation in seed by @dekkers in minvws/nl-kat-boefjes#46
- Updated dependabot config by @ammar92 in minvws/nl-kat-boefjes#70
New Contributors
- @Rieven made their first contribution in minvws/nl-kat-boefjes#37
- @Darwinkel made their first contribution in minvws/nl-kat-boefjes#44
Full Changelog: minvws/nl-kat-boefjes@v1.4.0...v1.5.0rc2
Rocky
What's Changed
- Release 1.4.0 by @noamblitz in minvws/nl-kat-rocky#75
- add missing dmarc, dkim and spf findings by @noamblitz in minvws/nl-kat-rocky#35
- Fix ignored flake8 errors and change tests to pytest by @dekkers in minvws/nl-kat-rocky#87
- fix boefje settings forms for integer fields by @noamblitz in minvws/nl-kat-rocky#91
- untested but adding cached network reference support to csv importer. by @underdarknl in minvws/nl-kat-rocky#95
- fix flake8 error on katalogus forms by @underdarknl in minvws/nl-kat-rocky#99
- Release 1.4 by @dekkers in minvws/nl-kat-rocky#102
- Run pytest in CI by @dekkers in minvws/nl-kat-rocky#113
- Bump json5 from 2.2.1 to 2.2.3 by @dependabot in minvws/nl-kat-rocky#103
- feature(scan level filter): implement by @Lisser in minvws/nl-kat-rocky#82
- CSV import introspection by @ammar92 in minvws/nl-kat-rocky#98
- Change Organization model and add API by @dekkers in minvws/nl-kat-rocky#109
- Feature/scan profile filter by @Lisser in minvws/nl-kat-rocky#83
- [Feature] [#C86] Add certificate expiry check by @zcrt in minvws/nl-kat-rocky#88
- Updated file structure for css files within rocky. by @HeleenSG in minvws/nl-kat-rocky#122
- fix(ooi_report): take description from finding_type if None in Finding by @Lisser in minvws/nl-kat-rocky#118
- Update landing_page.html by @ring-ring-ring in minvws/nl-kat-rocky#124
- Added tag styles and colors. Updated to manon v1.1.5 by @HeleenSG in minvws/nl-kat-rocky#133
- Add support for XTDB multinode by @dekkers in minvws/nl-kat-rocky#123
- Add task list params by @jpbruinsslot in minvws/nl-kat-rocky#89
- Fix scan history template artifact by @jpbruinsslot in minvws/nl-kat-rocky#135
- [OpenKAT v1.5] Fix boefje detail checkboxes function at scan oois by @Rieven in minvws/nl-kat-rocky#111
- Feature/multi select by @Donnype in minvws/nl-kat-rocky#86
- Fix/scan levels by @Donnype in minvws/nl-kat-rocky#154
- [OpenKAT v1.5] Multi XTDB and organizations management through organization code as slug by @Rieven in minvws/nl-kat-rocky#115
- Ignore minor updates by @ammar92 in minvws/nl-kat-rocky#155
- refactor raising of scanprofile by @...
v1.4.0
Summary
OpenKAT will be even more open:
OpenKAT has been open since summer 2022, with public releases after an internal release process. OpenKAT 1.4.0 is the last release that went through that process as we've moved to a fully open development process. We're one of the first government backed OSS projects to take this step and everyone is invited to join in.
The OpenKAT community is growing with contributions from within the healthcare sector, general non-profits and government suppliers. The uptake has been good and with each release the software becomes more mature and easier to use. We welcome everyone and look forward to working together on OpenKAT.
Features:
This release has several features, mostly aimed at stability and usability of the system. Several items are noteworthy:
-
Mime-types have been introduced to decouple boefjes and whiskers, our scanners and normalizers. It opens up a world of possibilities for new modules for OpenKAT.
-
Queuing in Mula, the scheduler, has been improved.
-
Pre-commit workflow: the new workflow will help anyone who wishes to contribute to OpenKAT, bringing the developers on the team and everyone else on the same level
-
Debian packages are now auto-generated for each module based on the latest release. We're working on a repository to make 'apt install openkat' a reality.
IMPORTANT
Add DATABASE_MIGRATION=true your .env to run all migrations on building.
Full Changelog
Coordination
What's Changed
- Fix table formatting by @dekkers in #72
- Add make down command by @helmo in #68
- Fix docker volume name in cleanup task by @helmo in #69
- Trigger compile on
mainbranch by @Darwinkel in #95 - Add missing keiko in Makefile and docker-compose by @dekkers
- Release 1.3.0 by @dekkers
- Add workflow and dockerfile for a debian build image by @errieman
- add git to build images to allow installation of octopoes via git by @errieman
- add ssh client to debian build image by @errieman
- Add more build dependencies to Debian Dockerfile by @dekkers
- Add missing KATALOGUS_API for Octopoes by @dekkers
- Also update Ubuntu dependencies by @dekkers
- Initial pre-commit, Sphinx, and GitHub Pages pipeline by @Darwinkel
New Contributors
Full Changelog: v1.3.0...v1.4.0
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- Fix toggle not working at ooi detail by @Rieven
- Bump toolz from 0.11.1 to 0.12.0 by @dependabot
- Bump psycopg2-binary from 2.8.6 to 2.9.5 by @dependabot
- Bump reportlab from 3.6.6 to 3.6.12 by @dependabot
- Bump colorama from 0.4.5 to 0.4.6 by @dependabot
- Bump requests from 2.26.0 to 2.28.1 by @dependabot
- Health link in footer by @Rieven
- Add total objects at all objects list by @Rieven
- Adds setting on organization member to assign and acknowledge OOI clearance levels on redteamers by @TwistMeister
- Bump urllib3 from 1.26.5 to 1.26.12 by @dependabot
- Bump python-dotenv from 0.19.0 to 0.21.0 by @dependabot
- Bump mypy from 0.971 to 0.991 by @dependabot
- Bump idna from 3.3 to 3.4 by @dependabot
- Bump qrcode from 6.1 to 7.3.1 by @dependabot
- Merge release back to develop by @dekkers
- Bump certifi from 2022.6.15 to 2022.9.24 by @dependabot
- Add scan levels to dev account in setup by @TwistMeister
- Improve onboarding "setup accounts" flow by @TwistMeister
- chore:upgraded to manon v1.1.0 by @HeleenSG
- Fixes task list download URL so it downloads zip with meta and RAW file by @TwistMeister
- Update stand alone checkboxes in forms in onboarding flow, to comply with new manon version by @TwistMeister
- Improve text "Got it, generate my report" to "start scanning" by @TwistMeister
- Bump vulture from 2.5 to 2.6 by @dependabot
- Bump djlint from 1.12.1 to 1.19.7 by @dependabot
- Revert "Bump djlint from 1.12.1 to 1.19.7" by @ammar92
- Bump pytz from 2021.1 to 2022.6 by @dependabot
- Set button states and heading styles by @TwistMeister
- Fix broken dashboard by @Rieven
- Move templates and locale directory inside rocky dir by @errieman
- Rocky deb package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Feature/support download multiple raw files by @Donnype
- fix: Updated to manon v1.1.1 which includes the footer fix by @HeleenSG
- Fix normalizers task list by @noamblitz
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
- Fix: re-use uuid by @noamblitz
- bump octopoes to 2.8.0 by @noamblitz
- Upgraded manon version by @HeleenSG
- Fixed: qr-code alignment and form message alignment by @HeleenSG
- Upgraded to manon v1.1.3 by @HeleenSG
- Fix setuptools version scheme by @dekkers in minvws/nl-kat-rocky#90
- Fix use of git tag in Debian package workflow by @dekkers in minvws/nl-kat-rocky#92
New Contributors
- @jpbruinsslot made their first contribution
Full Changelog: minvws/nl-kat-rocky@v1.3.0...v1.4.0
Boefjes
What's Changed
- fix env file by @underdarknl in minvws/nl-kat-boefjes#16
- LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#19
- Censys by @ammar92 in minvws/nl-kat-boefjes#20
- [log4shell] [#14] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#25
- [bug] [shodan] log no info instead of failing boefje by @zcrt in minvws/nl-kat-boefjes#22
- RE: [log4shell] [#14] [#25] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#28
- fixes, including formatting by @Lisser in minvws/nl-kat-boefjes#31
- Implement pre-commit hooks and some improvements by @ammar92
- Bump docker from 5.0.2 to 6.0.1 by @dependabot
- Bump jsonschema from 4.16.0 to 4.17.0 by @dependabot
- Bump sqlalchemy from 1.4.42 to 1.4.43 by @dependabot
- Bump click from 8.0.3 to 8.1.3 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- Fix fierce error handling by @Lisser
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump sqlalchemy from 1.4.43 to 1.4.44 by @dependabot
- Bump fastapi from 0.86.0 to 0.87.0 by @dependabot
- Add covers for: webpage analysis, greenhosting, leakIX and many new ports open by @TwistMeister
- Fix requirements by @dekkers
- Merge release back to develop by @dekkers
- Updates pre-commit pipeline by @ammar92
- Debian package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Fix alembic invocation by @dekkers
- Feature/multiple raw returns by @Donnype
- Feature/multiple raw returns by @noamblitz
- Refactor: prepare-for-containerization support by @Donnype
- Fix setuptools version scheme by @dekkers in minvws/nl-kat-boefjes#42
- Debian package fixes by @dekkers in minvws/nl-kat-boefjes#45
New Contributors
- @TwistMeister made their first contribution
- @jpbruinsslot made their first contribution
- @underdarknl made their first contribution in minvws/nl-kat-boefjes#16
- @zcrt made their first contribution in minvws/nl-kat-boefjes#19
- @ammar92 made their first contribution in minvws/nl-kat-boefjes#20
- @Lisser made their first contribution in minvws/nl-kat-boefjes#31
Full Changelog: minvws/nl-kat-boefjes@v1.3.0...v1.4.0
Octopoes
What's Changed
- Add pytest, show test coverage and add more tests by @dekkers
- Replace the CI pipeline with
pre-commitby @Darwinkel - Feature/robot improvements by @Lisser
- chore(robot): explicit host mapping in docker compose by @Lisser
- Bump setuptools from 65.5.0 to 65.5.1 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- chore(dependencies): fix dependencies by @Lisser
- Feature/pre commit compliance by @Lisser
- refactor(*): comply to vulture by @Lisser
- Fix pushing the scan profile increments events to the scheduler by @Lisser
- Updates pre-commit pipeline by @ammar92
- Merge release back to develop by @dekkers
- Add mypy pre-commit stubs by @Darwinkel
- Run unit ...
v1.4.0rc2
v1.4.0rc1
Summary
OpenKAT will be even more open:
OpenKAT has been open since this years' summer, with public releases after an internal release process. OpenKAT 1.4.0 is the last release that went through that process as we've moved to a fully open development process. We're one of the first government backed OSS projects to take this step and everyone is invited to join in.
The OpenKAT community is growing with contributions from within the healthcare sector, general non-profits and government suppliers. The uptake has been good and with each release the software becomes more mature and easier to use. We welcome everyone and look forward to working together on OpenKAT.
Features:
This release has several features, mostly aimed at stability and usability of the system. Several items are noteworthy:
-
Mime-types have been introduced to decouple boefjes and whiskers, our scanners and normalizers. It opens up a world of possibilities for new modules for OpenKAT.
-
Queuing in Mula, the scheduler, has been improved.
-
Pre-commit workflow: the new workflow will help anyone who wishes to contribute to OpenKAT, bringing the developers on the team and everyone else on the same level
-
Debian packages are now auto-generated for each module based on the latest release. We're working on a repository to make 'apt install openkat' a reality.
IMPORTANT
Add DATABASE_MIGRATION=true your .env to run all migrations on building.
Full Changelog
Coordination
What's Changed
- Add missing keiko in Makefile and docker-compose by @dekkers
- Release 1.3.0 by @dekkers
- Add workflow and dockerfile for a debian build image by @errieman
- add git to build images to allow installation of octopoes via git by @errieman
- add ssh client to debian build image by @errieman
- Add more build dependencies to Debian Dockerfile by @dekkers
- Add missing KATALOGUS_API for Octopoes by @dekkers
- Also update Ubuntu dependencies by @dekkers
- Initial pre-commit, Sphinx, and GitHub Pages pipeline by @Darwinkel
Full Changelog: v1.3.0...v1.4.0rc1
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- Fix toggle not working at ooi detail by @Rieven
- Bump toolz from 0.11.1 to 0.12.0 by @dependabot
- Bump psycopg2-binary from 2.8.6 to 2.9.5 by @dependabot
- Bump reportlab from 3.6.6 to 3.6.12 by @dependabot
- Bump colorama from 0.4.5 to 0.4.6 by @dependabot
- Bump requests from 2.26.0 to 2.28.1 by @dependabot
- Health link in footer by @Rieven
- Add total objects at all objects list by @Rieven
- Adds setting on organization member to assign and acknowledge OOI clearance levels on redteamers by @TwistMeister
- Bump urllib3 from 1.26.5 to 1.26.12 by @dependabot
- Bump python-dotenv from 0.19.0 to 0.21.0 by @dependabot
- Bump mypy from 0.971 to 0.991 by @dependabot
- Bump idna from 3.3 to 3.4 by @dependabot
- Bump qrcode from 6.1 to 7.3.1 by @dependabot
- Merge release back to develop by @dekkers
- Bump certifi from 2022.6.15 to 2022.9.24 by @dependabot
- Add scan levels to dev account in setup by @TwistMeister
- Improve onboarding "setup accounts" flow by @TwistMeister
- chore:upgraded to manon v1.1.0 by @HeleenSG
- Fixes task list download URL so it downloads zip with meta and RAW file by @TwistMeister
- Update stand alone checkboxes in forms in onboarding flow, to comply with new manon version by @TwistMeister
- Improve text "Got it, generate my report" to "start scanning" by @TwistMeister
- Bump vulture from 2.5 to 2.6 by @dependabot
- Bump djlint from 1.12.1 to 1.19.7 by @dependabot
- Revert "Bump djlint from 1.12.1 to 1.19.7" by @ammar92
- Bump pytz from 2021.1 to 2022.6 by @dependabot
- Set button states and heading styles by @TwistMeister
- Fix broken dashboard by @Rieven
- Move templates and locale directory inside rocky dir by @errieman
- Rocky deb package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Feature/support download multiple raw files by @Donnype
- fix: Updated to manon v1.1.1 which includes the footer fix by @HeleenSG
- Fix normalizers task list by @noamblitz
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
- Fix: re-use uuid by @noamblitz
- bump octopoes to 2.8.0 by @noamblitz
- Upgraded manon version by @HeleenSG
- Fixed: qr-code alignment and form message alignment by @HeleenSG
- Upgraded to manon v1.1.3 by @HeleenSG
New Contributors
- @jpbruinsslot made their first contribution
Full Changelog: minvws/nl-kat-rocky@v1.3.0...v1.4.0rc1
Boefjes
What's Changed
- fix env file by @underdarknl in minvws/nl-kat-boefjes#16
- LeakIX handle HTTP 204 response by @zcrt in minvws/nl-kat-boefjes#19
- Censys by @ammar92 in minvws/nl-kat-boefjes#20
- [log4shell] [#14] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#25
- [bug] [shodan] log no info instead of failing boefje by @zcrt in minvws/nl-kat-boefjes#22
- RE: [log4shell] [#14] [#25] make REPLY_FQDN variable by @zcrt in minvws/nl-kat-boefjes#28
- fixes, including formatting by @Lisser in minvws/nl-kat-boefjes#31
- Implement pre-commit hooks and some improvements by @ammar92
- Bump docker from 5.0.2 to 6.0.1 by @dependabot
- Bump jsonschema from 4.16.0 to 4.17.0 by @dependabot
- Bump sqlalchemy from 1.4.42 to 1.4.43 by @dependabot
- Bump click from 8.0.3 to 8.1.3 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- Fix fierce error handling by @Lisser
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump sqlalchemy from 1.4.43 to 1.4.44 by @dependabot
- Bump fastapi from 0.86.0 to 0.87.0 by @dependabot
- Add covers for: webpage analysis, greenhosting, leakIX and many new ports open by @TwistMeister
- Fix requirements by @dekkers
- Merge release back to develop by @dekkers
- Updates pre-commit pipeline by @ammar92
- Debian package with venv by @errieman
- Changes to support new scheduler setup by @jpbruinsslot
- Fix alembic invocation by @dekkers
- Feature/multiple raw returns by @Donnype
- Feature/multiple raw returns by @noamblitz
- Refactor: prepare-for-containerization support by @Donnype
New Contributors
- @TwistMeister made their first contribution
- @jpbruinsslot made their first contribution
- @underdarknl made their first contribution in minvws/nl-kat-boefjes#16
- @zcrt made their first contribution in minvws/nl-kat-boefjes#19
- @ammar92 made their first contribution in minvws/nl-kat-boefjes#20
- @Lisser made their first contribution in minvws/nl-kat-boefjes#31
Full Changelog: minvws/nl-kat-boefjes@v1.3.0...v1.4.0rc1
Octopoes
What's Changed
- Add pytest, show test coverage and add more tests by @dekkers
- Replace the CI pipeline with
pre-commitby @Darwinkel - Feature/robot improvements by @Lisser
- chore(robot): explicit host mapping in docker compose by @Lisser
- Bump setuptools from 65.5.0 to 65.5.1 by @dependabot
- Bump fastapi from 0.85.1 to 0.86.0 by @dependabot
- chore(dependencies): fix dependencies by @Lisser
- Feature/pre commit compliance by @Lisser
- refactor(*): comply to vulture by @Lisser
- Fix pushing the scan profile increments events to the scheduler by @Lisser
- Updates pre-commit pipeline by @ammar92
- Merge release back to develop by @dekkers
- Add mypy pre-commit stubs by @Darwinkel
- Run unit tests with supported Python versions by @Darwinkel
- feat(scan_profiles): send all mutation events to scheduler by @Lisser
- feat(api): implement object filtering based on scan level by @Lisser
- fix(types): implement some missing types by @Lisser
- chore(deps): update by @Lisser
- Initial Sphinx/autodoc support by @Darwinkel
- Potential fix for Pages deploy by @Darwinkel
- Fix trailing spaces in diagrams by @Darwinkel
- v2 architectural design document by @Lisser
- v2 -> v3 by @Lisser
- Ship venv with debian package by @errieman
- Remove redundant artifact upload by @Darwinkel
- default 0 for filtering by scan level by @Lisser
- Bump certifi from 2022.9.24 to 2022.12.7 by @dependabot
New Contributors
- @Darwinkel made their first contribution
- @dependabot made their first contribution
Full Changelog: https://github.com/minvws/nl-kat-octopoes/co...
v1.3.0
Summary
The 1.3.0 release of OpenKAT introduces new features and small fixes. The most interesting new elements are the way boefjes and normalizers are structured and the way settings are handled. Boefjes and normalizers are decoupled, to allow more matches between boefjes, normalizers and objects. Settings for boefjes have become available through the front end of OpenKAT. Also, OpenKAT has become easier to package and install.
IMPORTANT
Celery beat had been added to the Octopoes worker and the command used to start the Octopoes worker has been changed because of this. There needs to be one Octopoes worker that runs Celery beat to schedule tasks and there can also be only one worker with celery beat. For containers we added entrypoints that can handle any future changes, see https://github.com/minvws/nl-kat-coordination/blob/main/docs/containers.md#container-commands and https://github.com/minvws/nl-kat-coordination/blob/main/docker-compose.release-example.yml
Settings for boefjes
Building on the changes that were made in the previous release, from now on boefjes can have mandatory settings. For example, before enabling Shodan, you must provide the API key in the katalogus settings. This removes the need to place all boefje settings in a config file.
Plugin directory restructure
Boefjes and normalizers are now fully decoupled. This means that a plugin directory can now have as many subdirectories with boefjes and normalizers. For a boefjes or normalizer to be found, it has to have an __init__.py, a boefje.json or normalizer.json and a file with a run method. This sets a big step towards "package-ability" of plugins. On the downside, your previously created boefjes and normalizers need a minor update. The manual on the wiki has been adapted to reflect these changes and will help you make the transition.
Pagination of Octopoes
Before, when loading the objects list, all OOIs were returned. This created performance issues, and therefore we added API side pagination to octopoes.
Full Changelog
Coordination
What's Changed
- apply changes to octopoes worker by @Lisser
Rocky
What's Changed
- Download PDF Robot Flow by @reincode050
- Unify and update linter configs by @reincode050
- fix element not found by @Rieven
- Fix/styling checkboxes treeview by @Rieven
- add groups field to admin by @noamblitz
- support octopoes SCM versioning in docker dev setup by @errieman
- Improve User views in admin by @dekkers
- make rocky compatible with new scan profile model by @Lisser
- change name to id in normalizer task list by @noamblitz
- Create dependabot.yml by @sigio
- Update requirements.txt by @underdarknl
- Bump pylint from 2.13 to 2.15.5 by @dependabot
- build octopoes in main image by @errieman
- bump octopoes to 2.6.2 by @noamblitz
- accept jpg from katalogus by @noamblitz
- Bump django-formtools from 2.3 to 2.4 by @dependabot
- Bump sqlparse from 0.4.2 to 0.4.3 by @dependabot
- Bump uwsgi from 2.0.20 to 2.0.21 by @dependabot
- Required/mandatory settings for Boefjes by @Rieven
- perf(octopoes): improve pagination by @Lisser
- Fix for getting consumable oois at plugin detail by @Rieven
- Change date to time for checking OOI in future by @Rieven
- Translations before release by @Rieven
- Bump soupsieve from 2.2.1 to 2.3.2.post1 by @dependabot
- Bump phonenumbers from 8.12.30 to 8.12.57 by @dependabot
- Bump xhtml2pdf from 0.2.5 to 0.2.8 by @dependabot
- update yarn packages by @Rieven
Boefjes
What's Changed
- update instructions for running boefjes by @errieman
- Feature/json boefje definitions by @Donnype
- support octopoes SCM versioning in docker dev setup by @errieman
- Expose environment keys (mandatory fields) by @Donnype
- Feature/recursively find plugins by @Donnype
- Feature/module to path by @Donnype
- Discovery fix by @Donnype
- add correct port and address to cert boefje by @noamblitz
- Remove redundant API key notices from description by @noamblitz
- Create dependabot.yml by @sigio
- build octopoes in main image by @errieman
- fix(binaryedge): rename ooi_type to object_type by @Lisser
- bump octopoes to 2.6.2 by @noamblitz
- optimize katalogus images by @noamblitz
- Env vars globally by @sigio
- Bump sqlalchemy from 1.4.32 to 1.4.42 by @dependabot
- Bump fastapi from 0.75.0 to 0.85.1 by @dependabot
- Bump uvicorn from 0.17.5 to 0.19.0 by @dependabot
- Bump pytest from 6.2.5 to 7.2.0 by @dependabot
- revert id to name change by @noamblitz
- get api keys from katalogus api plugin settings by @noamblitz
- Bump dnspython from 2.1.0 to 2.2.1 by @dependabot
- Bump black from 21.9b0 to 22.10.0 by @dependabot
- Bump psycopg2-binary from 2.9.3 to 2.9.5 by @dependabot
- Bump alembic from 1.7.7 to 1.8.1 by @dependabot
- Bump requests from 2.27.1 to 2.28.1 by @dependabot
- Fix requirements for requests and dispython by @ammar92
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution
Octopoes
What's Changed
- fix(retirejs): avoid indexerror by @Lisser
- Minor improvements and optimalisation for scanprofile recalculate by @dekkers
- Release 2.6.0 by @Lisser
- refactor(list_ooi): add count to result by @Lisser
- Create pull_request_template.md by @Lisser
- Automated update to workflows: use checkout@v3 by @sigio
- fix packaging requirement by @noamblitz
Mula
What's changed
- Merge pull request #130 from minvws/feat/monitor-org-timeout
- Access queues directly from schedulers dict
Bytes
What's Changed
- Bump mako from 1.2.0 to 1.2.2 by @dependabot
- Hotfix/rename name to by @Donnype
- Create dependabot.yml by @sigio
- Automated update to workflows: use checkout@v3 by @sigio
New Contributors
- @dependabot made their first contribution