pkg icon indicating copy to clipboard operation
pkg copied to clipboard

Experimental changes to disable SQLite3 async writes and use WAL mode instead

Open cgull opened this issue 4 years ago • 0 comments

This is a so-far-experimental change to improve pkg's sqlite3 reliability, and quite possibly performance too. This has been lightly tested on a single FreeBSD machine.

Text copied from #2007 and slightly updated:

At the top of pkgdb_begin_solver():

	const char solver_sql[] = ""
		"PRAGMA synchronous = OFF;"
		"PRAGMA journal_mode = MEMORY;"
		"BEGIN TRANSACTION;";

and pkgdb_end_solver() restores the SQLite3 defaults.

This tells SQLite3 to not fsync(), and not to maintain any transaction log or any other recovery mechanism-- so if the system crashes, the database may be corrupted, inconsistent, and unrecoverable.

I'd like to suggest that this isn't good, and we should do something that at least gets us a consistent database after a crash.

Maybe we should use journal_mode=WAL and synchronous=NORMAL; that seems a better match to what pkg needs (the database is consistent but not durable). WAL mode is also supposedly faster and does much less fsync(). The locking is also said to be finer-grained, too. I haven't really used SQLite in situations where these changes might show, and it doesn't show any performance changes with pkg on my typical developer desktop FreeBSD system. WAL is persistent, we can set it once when we create the database.

Downsides:

  • WAL creates/opens 1 or 2 extra files (log file and SHM file), which might conflict with sandboxing.
  • WAL might not work well with pkg running on a single system using NFS, because of locking and atomicity issues (though the functionality and problems may be different than with SQL's logging mode, i.e. WAL mode might work in certain places where logging mode doesn't, or vice versa). It definitely won't work with multiple client machines sharing a single database because it uses SHM (I hope that's explicitly prohibited for pkg anyway, trying to use a single /var/db/pkg/ for multiple systems can only lead to pain, inconsistency, and broken systems).

cgull avatar Nov 24 '21 23:11 cgull