Database recovery failure under a few restarts in a row.
What happened?
Sharkey is failing to send posts or reactions from the user side.
What did you expect to happen?
Sharkey is able to send posts & reactions.
Version
Stable latest
Instance
What type of issue is this?
server-side
What browser are you using? (Client-side issues only)
LibreWolf 132.0.2-1
What operating system are you using? (Client-side issues only)
Alpine Linux v3.20
How do you deploy Sharkey on your server? (Server-side issues only)
through Docker compose.
What operating system are you using? (Server-side issues only)
Debian 12
Relevant log output (Please copy and paste any relevant log output. You can find your log by inspecting the page, and going to the "console" tab. This will be automatically formatted into code, so no need for backticks.)
web-1 | ERR 4 [queue inbox] failed(QueryFailedError: invalid page in block 48138 of relation base/16384/16507) id=1373 attempts=8/8 age=262m activity=https://kitsunes.club/notes/a1641xmuy7/activity {
web-1 | job: {
web-1 | name: undefined,
web-1 | info: 'id=1373 attempts=8/8 age=262m',
web-1 | failedReason: 'invalid page in block 48138 of relation base/16384/16507',
web-1 | data: { activity: [Object], signature: [Object] }
web-1 | },
web-1 | e: {
web-1 | stack: 'QueryFailedError: invalid page in block 48138 of relation base/16384/16507\n' +
web-1 | ' at PostgresQueryRunner.query (/sharkey/node_modules/.pnpm/typeorm@0.3.20_ioredis@5.4.1_pg@8.13.0/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n' +
web-1 | ' at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
web-1 | ' at async InsertQueryBuilder.execute (/sharkey/node_modules/.pnpm/typeorm@0.3.20_ioredis@5.4.1_pg@8.13.0/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n' +
web-1 | ' at async NoteCreateService.insertNote (file:///sharkey/packages/backend/built/core/NoteCreateService.js:454:17)\n' +
web-1 | ' at async NoteCreateService.create (file:///sharkey/packages/backend/built/core/NoteCreateService.js:372:22)\n' +
web-1 | ' at async ApNoteService.createNote (file:///sharkey/packages/backend/built/core/activitypub/models/ApNoteService.js:303:20)\n' +
web-1 | ' at async ApNoteService.resolveNote (file:///sharkey/packages/backend/built/core/activitypub/models/ApNoteService.js:563:20)\n' +
web-1 | ' at async ApNoteService.createNote (file:///sharkey/packages/backend/built/core/activitypub/models/ApNoteService.js:228:40)\n' +
web-1 | ' at async ApInboxService.createNote (file:///sharkey/packages/backend/built/core/activitypub/ApInboxService.js:406:13)\n' +
web-1 | ' at async ApInboxService.create (file:///sharkey/packages/backend/built/core/activitypub/ApInboxService.js:383:13)\n' +
web-1 | ' at async ApInboxService.performOneActivity (file:///sharkey/packages/backend/built/core/activitypub/ApInboxService.js:162:20)\n' +
web-1 | ' at async ApInboxService.performActivity (file:///sharkey/packages/backend/built/core/activitypub/ApInboxService.js:147:22)\n' +
web-1 | ' at async InboxProcessorService.process (file:///sharkey/packages/backend/built/queue/processors/InboxProcessorService.js:195:28)\n' +
web-1 | ' at async Worker.processJob (/sharkey/node_modules/.pnpm/bullmq@5.13.2/node_modules/bullmq/dist/cjs/classes/worker.js:455:28)\n' +
web-1 | ' at async Worker.retryIfFailed (/sharkey/node_modules/.pnpm/bullmq@5.13.2/node_modules/bullmq/dist/cjs/classes/worker.js:640:24)',
web-1 | message: 'invalid page in block 48138 of relation base/16384/16507',
web-1 | name: 'QueryFailedError'
web-1 | }
web-1 | }
web-1 | query is slow: UPDATE "instance" SET "notesCount" = "notesCount" + 1 WHERE "id" = $1 -- PARAMETERS: ["9r7u06okswo1003b"]
web-1 | execution time: 397Contribution Guidelines By submitting this issue, you agree to follow our Contribution Guidelines
- I agree to follow this project's Contribution Guidelines
- I have searched the issue tracker for similar issues, and this is not a duplicate.
Activity
This is unrelated to Sharkey - your Postgres database has become corrupted. You can restore from a backup, or try to research other recovery options. I'm unfortunately not familiar enough with Postgres data recovery to provide any advice.
Maybe this will help? https://stackoverflow.com/questions/62422298/postgresql-pg-dump-for-invalid-page-in-block-database-does-not-work-properly
added invalid label
I don't like that approach. At the point of DB failure it is expected people manually intervene. The database can be broken for a variety of reasons and it's simply infeasible to ask us to factor in each of them. Besides, what happens when the automatic attempt to fix the database fails? Even if the solution was to zero pages what if that's not the correct approach for other problems and it zeros the pages anyway? I say this as someone who has watched chkdsk https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/chkdsk spectacularly fail and make my data nearly irrecoverable instead of actually fixing the corrupted NTFS partition. Once you violate the integrity of the DB there's not much we can do. We rely a lot on Postgres to keep us sane (postgres enforces our constraints that we specify). This sort of automatic recovery would be on the DBMS (Database Management Software) instead of on the application side.
