Skip to content

Commit 28e5d1a

Browse files
empusMrIron-no
empus
authored and
MrIron-no
committed
feat(migrations): Add automatic DB migrations:
- Automatically applied from ./mod.{module}/migrations/*.sql files - Run gnuworld to apply - Applies migrations persisted in gnuworld_migrations tables for each module - See doc/README.migrations for details
1 parent b26c779 commit 28e5d1a

19 files changed

Lines changed: 875 additions & 6 deletions

File tree

‎Makefile.am‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,39 @@ noinst_LTLIBRARIES =
99

1010
pkgdata_DATA = bin/server_command_map
1111

12+
# Install migration files for configured modules
13+
if COND_MODCSERVICE
14+
install-data-local: install-cservice-migrations
15+
endif
16+
17+
if COND_MODCCONTROL
18+
install-data-local: install-ccontrol-migrations
19+
endif
20+
21+
if COND_MODDRONESCAN
22+
install-data-local: install-dronescan-migrations
23+
endif
24+
25+
if COND_MODOPENCHANFIX
26+
install-data-local: install-openchanfix-migrations
27+
endif
28+
29+
install-cservice-migrations:
30+
$(MKDIR_P) $(DESTDIR)$(prefix)/migrations/cservice
31+
cp -r mod.cservice/migrations/* $(DESTDIR)$(prefix)/migrations/cservice/
32+
33+
install-ccontrol-migrations:
34+
$(MKDIR_P) $(DESTDIR)$(prefix)/migrations/ccontrol
35+
cp -r mod.ccontrol/migrations/* $(DESTDIR)$(prefix)/migrations/ccontrol/
36+
37+
install-dronescan-migrations:
38+
$(MKDIR_P) $(DESTDIR)$(prefix)/migrations/dronescan
39+
cp -r mod.dronescan/migrations/* $(DESTDIR)$(prefix)/migrations/dronescan/
40+
41+
install-openchanfix-migrations:
42+
$(MKDIR_P) $(DESTDIR)$(prefix)/migrations/openchanfix
43+
cp -r mod.openchanfix/migrations/* $(DESTDIR)$(prefix)/migrations/openchanfix/
44+
1245
EXTRA_DIST = bin/ccontrol.example.conf \
1346
bin/chanfix.example.conf \
1447
bin/clientExample.example.conf \

‎doc/README.migrations‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
GNUWorld Database Migration System
2+
===================================
3+
4+
Overview
5+
--------
6+
7+
The database migration system automatically tracks and applies SQL schema updates
8+
to GNUWorld module databases. When a gnuworld instance starts, it checks for any
9+
unapplied migration files and applies them automatically. If a migration fails to apply,
10+
the instance will exit with a clear error message.
11+
12+
This system ensures that:
13+
- Schema updates are applied automatically on startup
14+
- All instances of gnuworld stay in sync with schema requirements
15+
- Migrations are tracked persistently in the database
16+
- Failed migrations are clearly reported with error details
17+
18+
19+
How It Works
20+
------------
21+
22+
1. **gnuworld_migrations Table**
23+
Each module database contains a gnuworld_migrations table that tracks which
24+
migration files have been applied. The table has columns:
25+
- id: Unique identifier
26+
- module: Module name (e.g., "cservice", "ccontrol")
27+
- file: The SQL filename that was applied (e.g., "001_add_column.sql")
28+
- applied_at: ISO 8601 timestamp with timezone offset when the migration was applied
29+
(e.g., "2026-03-23 14:30:45.123456+00:00")
30+
31+
This table is created automatically on first startup if it doesn't exist.
32+
33+
2. **Migration Files**
34+
Migration files are stored in each module's migrations directory:
35+
- mod.cservice/migrations/
36+
- mod.ccontrol/migrations/
37+
- mod.dronescan/migrations/
38+
- mod.openchanfix/migrations/
39+
40+
Files must follow the naming convention: NNN_description.sql
41+
Where NNN is a three-digit number (001, 002, 003, etc.)
42+
43+
3. **Startup Check and Auto-Apply**
44+
When each module starts:
45+
a) Database connection is established
46+
b) Migration system checks for unapplied migrations
47+
c) If all migrations are applied: startup continues normally
48+
d) If any migrations are pending: they are applied automatically
49+
- Each migration file is read and executed
50+
- Successfully applied migrations are recorded in gnuworld_migrations
51+
e) If any migration fails: startup exits with an error message
52+
- Error message includes the failed migration name and SQL error details
53+
- No other migrations are attempted after a failure
54+
55+
4. **What If a Migration Fails?**
56+
If a migration fails to apply:
57+
a) The error message explains what went wrong
58+
b) The module fails to start
59+
c) You must fix the issue (database state or migration file)
60+
d) Restart gnuworld to retry
61+
62+
63+
Creating New Migration Files
64+
----------------------------
65+
66+
When you need to modify a module's database schema, create a migration file:
67+
68+
1. **Determine the next migration number**
69+
List existing migrations:
70+
ls mod.cservice/migrations/
71+
72+
If the last migration is 002_add_table.sql, the next migration is 003
73+
74+
2. **Create the migration file**
75+
Create: mod.cservice/migrations/003_add_column.sql
76+
77+
Example content:
78+
ALTER TABLE users ADD COLUMN IF NOT EXISTS new_field VARCHAR(100);
79+
CREATE INDEX IF NOT EXISTS idx_users_new_field ON users(new_field);
80+
81+
3. **Important: Make migrations idempotent**
82+
Always use:
83+
- CREATE TABLE IF NOT EXISTS
84+
- ALTER TABLE ... ADD COLUMN IF NOT EXISTS
85+
- CREATE INDEX IF NOT EXISTS
86+
- DROP TABLE IF EXISTS
87+
88+
This ensures migrations can be safely re-run without errors.
89+
90+
4. **Document the change**
91+
Add a comment at the top of the migration file explaining what changed:
92+
93+
-- Migration 003: Add new_field column to users table for feature X
94+
-- Purpose: Support tracking of user metadata
95+
ALTER TABLE users ADD COLUMN IF NOT EXISTS new_field VARCHAR(100);
96+
97+
5. **Test the migration**
98+
Apply it manually to verify it works:
99+
psql cservice < mod.cservice/migrations/003_add_column.sql
100+
101+
Verify the schema change:
102+
psql cservice -c "\d users"
103+
104+
105+
Applying Migrations Automatically (Default Behavior)
106+
----------------------------------------------------
107+
108+
Migrations are applied automatically when gnuworld starts. Simply restart gnuworld:
109+
110+
./gnuworld -f bin/GNUWorld.conf
111+
112+
The migration system will:
113+
1. Detect any unapplied migrations
114+
2. Apply them in order (001, 002, 003, etc.)
115+
3. Log each migration as it's applied
116+
4. Exit with an error if any migration fails
117+
118+
119+
Disabling Migration Files
120+
--------------------------------------------------
121+
122+
To prevent a migration file refrom running, change the *.sql extension. e.g.,
123+
124+
mv ./mod.{module}/migrations/001_update_timestamp.sql ./mod.{module}/migrations/001_update_timestamp.sql.tmp
125+
126+
127+
Verifying Applied Migrations
128+
-----------------------------
129+
130+
To check which migrations have been applied to a module database:
131+
132+
psql cservice -c "SELECT file FROM gnuworld_migrations WHERE module='cservice' ORDER BY applied_at;"
133+
134+
Expected output:
135+
file
136+
---------------------
137+
001_add_column.sql
138+
002_create_index.sql
139+
(2 rows)
140+
141+
142+
143+
Troubleshooting
144+
---------------
145+
146+
**Problem: gnuworld fails to start with a migration error**
147+
When a migration fails to apply, the error message will say:
148+
ERROR [MigrationChecker]: Failed to apply migration 'XXX_name.sql'...
149+
150+
Causes and solutions:
151+
- Syntax error in the migration file - fix the SQL in the file
152+
- Missing prerequisite (e.g., trying to add a column to a non-existent table) - fix the migration logic
153+
- Database permissions - ensure your database user has schema change permissions
154+
- Migration already partially applied - manually check the database state
155+
156+
To recover:
157+
1. Review the error message to understand what went wrong
158+
2. Fix either the migration file or the database state
159+
3. Restart gnuworld to retry the migration
160+
161+
**Problem: A migration file has a syntax error**
162+
If a migration file has an SQL syntax error:
163+
1. Fix the migration file directly (e.g., mod.cservice/migrations/001_name.sql)
164+
2. Restart gnuworld
165+
3. The migration will be retried and should succeed
166+
167+
**Problem: gnuworld doesn't apply my new migration file**
168+
Ensure:
169+
1. The file follows the naming convention: NNN_description.sql (e.g., 003_add_column.sql)
170+
2. The file is in the correct directory: mod.{module}/migrations/
171+
3. The file is readable by the gnuworld process
172+
4. Restart gnuworld - migrations are checked on startup
173+
174+
**Problem: Migration was applied but I want to undo it**
175+
The migration system does not support automated rollback. If you need to undo changes:
176+
1. Write a new migration that reverses the previous one (e.g., migration 004 undoes 003)
177+
2. Or manually undo the schema changes via psql
178+
3. If you manually undo via psql, restart gnuworld (it will not re-apply if recorded)
179+
180+
**Problem: Need to undo a migration (rollback)**
181+
The migration system does not support automated rollback. If you need to undo changes:
182+
1. Write a new migration that reverses the previous one
183+
2. Or manually undo the schema changes via SQL
184+
3. Then run a new migration to record the change
185+
186+
Example: If migration 003 added a column, write migration 004 to drop it:
187+
-- Migration 004: Revert column added in migration 003
188+
ALTER TABLE users DROP COLUMN IF EXISTS new_field;
189+
190+
191+
Best Practices
192+
--------------
193+
194+
1. **One logical change per migration file**
195+
Don't combine unrelated schema changes in a single migration.
196+
197+
2. **Test migrations locally first**
198+
Apply migrations to a test database before deploying to production.
199+
200+
3. **Use descriptive migration names**
201+
Bad: 001_update.sql
202+
Good: 001_add_user_preferences_table.sql
203+
204+
4. **Keep migrations small and focused**
205+
Easier to debug if something goes wrong.
206+
207+
5. **Document complex migrations**
208+
Add comments explaining why the change was made.
209+
210+
6. **Consider performance impact**
211+
Large table modifications on production systems may require special handling.
212+
Document any expected downtime or performance considerations.
213+
214+
7. **Coordinate deployments**
215+
If multiple gnuworld instances share a database, apply migrations before
216+
restarting instances to avoid conflicts.
217+
218+
8. **Back up before major migrations**
219+
Before applying significant schema changes, back up your database:
220+
pg_dump cservice > cservice_backup.sql
221+
222+
223+
224+
Questions or Issues?
225+
--------------------
226+
227+
If you encounter issues with the migration system:
228+
229+
1. Check the error message - it usually tells you exactly what's wrong
230+
2. Read the troubleshooting section above
231+
3. Review the migration file that's causing issues
232+
4. Check your database permissions
233+
5. Consult the GNUWorld IRC channel: #coder-com on Undernet
234+
235+
See also:
236+
- GNUWorld documentation: https://github.com/UndernetIRC/gnuworld
237+
- IRC support: irc.undernet.org #coder-com

‎include/client.h‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,9 @@
3737

3838
namespace gnuworld {
3939

40+
// Forward declarations
41+
class dbHandle;
42+
4043
/**
4144
* This is the public concrete base class that represents
4245
* a client that may connect to this server. To build a new
@@ -924,6 +927,25 @@ class xClient : public TimerHandler, public NetworkTarget {
924927
* Logger instance.
925928
*/
926929
std::unique_ptr<Logger> logger;
930+
931+
/**
932+
* Flag to track whether migrations have been checked for this module.
933+
* Prevents repeated checks on the same instance.
934+
*/
935+
bool migrationsChecked = false;
936+
937+
/**
938+
* Check database migrations for this module.
939+
* Verifies that all migration files have been applied to the database.
940+
* Must be called after database connection is established.
941+
*
942+
* @param moduleName The name of the module (e.g., "cservice", "ccontrol")
943+
* This is also used as the database name for migration tracking
944+
* @param db Pointer to the database connection handle
945+
* @return true if all migrations are applied, false otherwise
946+
* On failure, detailed error messages are logged to elog
947+
*/
948+
bool checkMigrationsAfterDBConnect(const std::string& moduleName, dbHandle* db);
927949
};
928950

929951
} // namespace gnuworld

‎libgnuworld/Makefile.am‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ libgnuworld_la_SOURCES = \
99
libgnuworld/ConnectionManager.cc \
1010
libgnuworld/EConfig.cc \
1111
libgnuworld/ELog.cc \
12+
libgnuworld/MigrationChecker.cc \
1213
libgnuworld/Signal.cc \
1314
libgnuworld/StringTokenizer.cc \
1415
libgnuworld/gThread.cc \
@@ -21,7 +22,9 @@ libgnuworld_la_LDFLAGS = $(threadLib)
2122

2223
libgnuworld_la_CXXFLAGS = \
2324
-I$(top_srcdir)/include \
24-
-I$(top_srcdir)/libgnuworld
25+
-I$(top_srcdir)/libgnuworld \
26+
-I$(top_srcdir)/libnotifier \
27+
-I$(top_srcdir)/db
2528

2629
EXTRA_DIST += \
2730
libgnuworld/Buffer.h \
@@ -30,6 +33,7 @@ EXTRA_DIST += \
3033
libgnuworld/ConnectionManager.h \
3134
libgnuworld/EConfig.h \
3235
libgnuworld/ELog.h \
36+
libgnuworld/MigrationChecker.h \
3337
libgnuworld/gThread.h \
3438
libgnuworld/ircd_chattr.h \
3539
libgnuworld/match.h \

0 commit comments

Comments
 (0)