## Deprecate and Remove the `~log` Table ### Summary We propose removing the per-schema `~log` table feature from DataJoint. This client-side audit logging mechanism has become obsolete and will be deprecated with no backward compatibility path. ### Background The `~log` table was designed to provide an audit trail of schema operations (table declarations, alterations, drops, and deletes) by storing event records directly in each schema's database. While well-intentioned, this approach no longer aligns with modern data management practices. ### Rationale for Removal **1. Server-side logging is the modern standard** Database platforms now provide robust, built-in audit capabilities: - MySQL Enterprise Audit, General Query Log, Binary Log - PostgreSQL `pgaudit` - Cloud-managed database audit logs (AWS RDS, GCP Cloud SQL, Azure) These server-side solutions capture *all* database activity regardless of client, providing complete coverage that client-side logging cannot achieve. **2. Incomplete and bypassable** The `~log` table only records operations performed through DataJoint. Any direct SQL access, other clients, or administrative tools bypass it entirely—creating a false sense of audit completeness. **3. Fragmented audit trail** Each schema maintains its own `~log` table, making centralized audit analysis difficult and providing no cross-schema visibility. **4. Users already have better alternatives** Modern data teams typically rely on: - Infrastructure observability platforms (Datadog, CloudWatch, Prometheus/Grafana) - Centralized logging systems (ELK stack, Splunk, Loki) - Database-native audit logs for compliance requirements **5. Maintenance burden** The log table grows indefinitely with no built-in rotation, cleanup, or retention policies. It adds storage overhead and schema clutter for minimal practical benefit. **6. Python `logging` module is sufficient** DataJoint already uses Python's standard `logging` module. We will ensure comprehensive logging coverage for all significant operations, allowing users to route logs to their preferred backends (files, syslog, cloud services, etc.) using standard Python logging configuration. ### Migration Path - The `~log` table will be removed with no backward compatibility - Existing `~log` tables will remain in user databases but will no longer be written to or maintained by DataJoint - Users relying on `~log` for audit purposes should migrate to database-level audit logging or external logging infrastructure - All operations previously logged to `~log` will be emitted through Python's `logging` module at appropriate log levels ### Affected Components - `datajoint.table.Log` class - `schema.log` property - Internal `_log()` calls in table declaration, alteration, deletion, and drop operations - `~log` table auto-creation on schema instantiation