Implement new Background Task queue
What does this MR do?
Implements a new primary queue type: "Background Task", which handles logic previously relegated to "fire-and-forget" async promises. This new implementation improves performance, is more reliable, and maintains system stability during bursts of heavy load. To complement the new queue, this PR also overhauls the "Collapsed Queue" implementation - making it faster and safe to use in multi-process deployments.
Background Task types:
-
update-user: updates a remote user over AP. -
update-featured: updates a remote user's featured notes over AP. -
update-instance: updates a remote instance over AP. -
update-user-tags: updates rankings for a user's featured hashtags. -
update-note-tags: updates rankings for hashtags used in a note. -
update-latest-note: updates "latest note" index after a note is created, edited, or deleted. -
post-deliver: updates stats and metrics after attempting to deliver an activity. -
post-inbox: updates stats and metrics after attempting to process an inbox activity. -
post-note: complete slow "post processing" tasks after a local or remote note is created or edited. -
post-suspend: pauses federation, notifications, and other automatic behavior after a user is suspended. -
post-unsuspend: restores federation, notifications, and other automatic behavior after a suspension is revoked. -
delete-file: removes a drive file, including the physical file and/or S3 blob. -
delete-ap-logs: removes all AP logs associated with a given URI(s).
Collapsed Queue types:
-
updateInstanceQueue: aggregates incremental federation stats and policy changes toinstanceentities. -
updateUserQueue: aggregates incremental stats touserentities. -
updateNoteQueue: aggregates incremental stats tonoteentities. -
updateAccessTokenQueue: accumulates activity foraccess_tokenentities. -
updateAntennaQueue: accumulates activity forantennaentities.
Additional performance improvements and micro-optimizations:
- Use
CacheServiceto fetch users in many more places. - Limit concurrency when performing parallel async
mapoperations. - Add caching to reduce database load of
get-online-users-count. - Remove unused API and websocket calls from admin dashboard
- Opportunistically use note.user relation in PollService.deliverQuestionUpdate
- Slightly reduce stalls when updating
MiInstance.
Additional stability and reliability improvements:
- Six cache sync after populating a user's banner.
- Synchronize collapsed queues across the cluster to avoid data races.
- Migrate to
InternalEventServiceinApPersonService. - Update
userProfileCachefromupdatePerson. - More efficient algorithm for identifying cascaded notes.
- Make
NoteDeleteService.deletesafe to call inside a loop. - Fix account deletion not triggering background tasks for cascaded note deletions.
-
awaitmore promises to improve stack traces. - Track federation-related promises to avoid data loss during restart.
Additional bug fixes:
- Fix incorrect counts for self-renotes and renotes by bots.
- Fix cascaded note deletion missing some notes.
- Fix stall in
EnableInstanceHOTUpdates1750217001651migration.
Additional technical changes:
- Add
immediateoption toNoteDeleteService. - Add
MiUser.lastFetchedFeaturedAt. - Fix typings of
resolveCollectionandresolveCollectionItems. - Fix typings of
Serialized<T>. - Add
CollapsedQueue.onPerformcallback. - Add
CollapsedQueue.deletemethod. - Return results from
CollapsedQueue.performAllNow. - Fix API schema for
admin/queue/jobsendpoint. - Add name and error logging to
CollapsedQueue. - Rename
noteEditRepositorytonoteEditsRepository, matching other repository names. - Remove useless async/await from charts.
Contribution Guidelines
By submitting this merge request, you agree to follow our Contribution Guidelines
-
I agree to follow this project's Contribution Guidelines -
I have made sure to test this merge request