Revamp post filtering system (#18058)
* Add model for custom filter keywords * Use CustomFilterKeyword internally Does not change the API * Fix /filters/edit and /filters/new * Add migration tests * Remove whole_word column from custom_filters (covered by custom_filter_keywords) * Redesign /filters Instead of a list, present a card that displays more information and handles multiple keywords per filter. * Redesign /filters/new and /filters/edit to add and remove keywords This adds a new gem dependency: cocoon, as well as a npm dependency: cocoon-js-vanilla. Those are used to easily populate and remove form fields from the user interface when manipulating multiple keyword filters at once. * Add /api/v2/filters to edit filter with multiple keywords Entities: - `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context` `keywords` - `FilterKeyword`: `id`, `keyword`, `whole_word` API endpoits: - `GET /api/v2/filters` to list filters (including keywords) - `POST /api/v2/filters` to create a new filter `keywords_attributes` can also be passed to create keywords in one request - `GET /api/v2/filters/:id` to read a particular filter - `PUT /api/v2/filters/:id` to update a new filter `keywords_attributes` can also be passed to edit, delete or add keywords in one request - `DELETE /api/v2/filters/:id` to delete a particular filter - `GET /api/v2/filters/:id/keywords` to list keywords for a filter - `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a filter - `GET /api/v2/filter_keywords/:id` to read a particular keyword - `PUT /api/v2/filter_keywords/:id` to edit a particular keyword - `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword * Change from `irreversible` boolean to `action` enum * Remove irrelevent `irreversible_must_be_within_context` check * Fix /filters/new and /filters/edit with update for filter_action * Fix Rubocop/Codeclimate complaining about task names * Refactor FeedManager#phrase_filtered? This moves regexp building and filter caching to the `CustomFilter` class. This does not change the functional behavior yet, but this changes how the cache is built, doing per-custom_filter regexps so that filters can be matched independently, while still offering caching. * Perform server-side filtering and output result in REST API * Fix numerous filters_changed events being sent when editing multiple keywords at once * Add some tests * Use the new API in the WebUI - use client-side logic for filters we have fetched rules for. This is so that filter changes can be retroactively applied without reloading the UI. - use server-side logic for filters we haven't fetched rules for yet (e.g. network error, or initial timeline loading) * Minor optimizations and refactoring * Perform server-side filtering on the streaming server * Change the wording of filter action labels * Fix issues pointed out by linter * Change design of “Show anyway” link in accordence to review comments * Drop “irreversible” filtering behavior * Move /api/v2/filter_keywords to /api/v1/filters/keywords * Rename `filter_results` attribute to `filtered` * Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer * Fix systemChannelId value in streaming server * Simplify code by removing client-side filtering code The simplifcation comes at a cost though: filters aren't retroactively applied anymore.
This commit is contained in:
Claire
GitHub
parent
5823ae70c4
commit
02851848e9
60 changed files with 1292 additions and 250 deletions
|
|
@ -247,6 +247,19 @@ module AccountInteractions
|
|||
account_pins.where(target_account: account).exists?
|
||||
end
|
||||
|
||||
def status_matches_filters(status)
|
||||
active_filters = CustomFilter.cached_filters_for(id)
|
||||
|
||||
filter_matches = active_filters.filter_map do |filter, rules|
|
||||
next if rules[:keywords].blank?
|
||||
|
||||
match = rules[:keywords].match(status.proper.searchable_text)
|
||||
FilterResultPresenter.new(filter: filter, keyword_matches: [match.to_s]) unless match.nil?
|
||||
end
|
||||
|
||||
filter_matches
|
||||
end
|
||||
|
||||
def followers_for_local_distribution
|
||||
followers.local
|
||||
.joins(:user)
|
||||
|
|
|
|||
|
|
@ -3,18 +3,22 @@
|
|||
#
|
||||
# Table name: custom_filters
|
||||
#
|
||||
# id :bigint(8) not null, primary key
|
||||
# account_id :bigint(8)
|
||||
# expires_at :datetime
|
||||
# phrase :text default(""), not null
|
||||
# context :string default([]), not null, is an Array
|
||||
# whole_word :boolean default(TRUE), not null
|
||||
# irreversible :boolean default(FALSE), not null
|
||||
# created_at :datetime not null
|
||||
# updated_at :datetime not null
|
||||
# id :bigint not null, primary key
|
||||
# account_id :bigint
|
||||
# expires_at :datetime
|
||||
# phrase :text default(""), not null
|
||||
# context :string default([]), not null, is an Array
|
||||
# created_at :datetime not null
|
||||
# updated_at :datetime not null
|
||||
# action :integer default(0), not null
|
||||
#
|
||||
|
||||
class CustomFilter < ApplicationRecord
|
||||
self.ignored_columns = %w(whole_word irreversible)
|
||||
|
||||
alias_attribute :title, :phrase
|
||||
alias_attribute :filter_action, :action
|
||||
|
||||
VALID_CONTEXTS = %w(
|
||||
home
|
||||
notifications
|
||||
|
|
@ -26,16 +30,20 @@ class CustomFilter < ApplicationRecord
|
|||
include Expireable
|
||||
include Redisable
|
||||
|
||||
enum action: [:warn, :hide], _suffix: :action
|
||||
|
||||
belongs_to :account
|
||||
has_many :keywords, class_name: 'CustomFilterKeyword', foreign_key: :custom_filter_id, inverse_of: :custom_filter, dependent: :destroy
|
||||
accepts_nested_attributes_for :keywords, reject_if: :all_blank, allow_destroy: true
|
||||
|
||||
validates :phrase, :context, presence: true
|
||||
validates :title, :context, presence: true
|
||||
validate :context_must_be_valid
|
||||
validate :irreversible_must_be_within_context
|
||||
|
||||
scope :active_irreversible, -> { where(irreversible: true).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()')) }
|
||||
|
||||
before_validation :clean_up_contexts
|
||||
after_commit :remove_cache
|
||||
|
||||
before_save :prepare_cache_invalidation!
|
||||
before_destroy :prepare_cache_invalidation!
|
||||
after_commit :invalidate_cache!
|
||||
|
||||
def expires_in
|
||||
return @expires_in if defined?(@expires_in)
|
||||
|
|
@ -44,22 +52,55 @@ class CustomFilter < ApplicationRecord
|
|||
[30.minutes, 1.hour, 6.hours, 12.hours, 1.day, 1.week].find { |expires_in| expires_in.from_now >= expires_at }
|
||||
end
|
||||
|
||||
def irreversible=(value)
|
||||
self.action = value ? :hide : :warn
|
||||
end
|
||||
|
||||
def irreversible?
|
||||
hide_action?
|
||||
end
|
||||
|
||||
def self.cached_filters_for(account_id)
|
||||
active_filters = Rails.cache.fetch("filters:v3:#{account_id}") do
|
||||
scope = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account_id: account_id }).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()'))
|
||||
scope.to_a.group_by(&:custom_filter).map do |filter, keywords|
|
||||
keywords.map! do |keyword|
|
||||
if keyword.whole_word
|
||||
sb = /\A[[:word:]]/.match?(keyword.keyword) ? '\b' : ''
|
||||
eb = /[[:word:]]\z/.match?(keyword.keyword) ? '\b' : ''
|
||||
|
||||
/(?mix:#{sb}#{Regexp.escape(keyword.keyword)}#{eb})/
|
||||
else
|
||||
/#{Regexp.escape(keyword.keyword)}/i
|
||||
end
|
||||
end
|
||||
[filter, { keywords: Regexp.union(keywords) }]
|
||||
end
|
||||
end.to_a
|
||||
|
||||
active_filters.select { |custom_filter, _| !custom_filter.expired? }
|
||||
end
|
||||
|
||||
def prepare_cache_invalidation!
|
||||
@should_invalidate_cache = true
|
||||
end
|
||||
|
||||
def invalidate_cache!
|
||||
return unless @should_invalidate_cache
|
||||
@should_invalidate_cache = false
|
||||
|
||||
Rails.cache.delete("filters:v3:#{account_id}")
|
||||
redis.publish("timeline:#{account_id}", Oj.dump(event: :filters_changed))
|
||||
redis.publish("timeline:system:#{account_id}", Oj.dump(event: :filters_changed))
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def clean_up_contexts
|
||||
self.context = Array(context).map(&:strip).filter_map(&:presence)
|
||||
end
|
||||
|
||||
def remove_cache
|
||||
Rails.cache.delete("filters:#{account_id}")
|
||||
redis.publish("timeline:#{account_id}", Oj.dump(event: :filters_changed))
|
||||
end
|
||||
|
||||
def context_must_be_valid
|
||||
errors.add(:context, I18n.t('filters.errors.invalid_context')) if context.empty? || context.any? { |c| !VALID_CONTEXTS.include?(c) }
|
||||
end
|
||||
|
||||
def irreversible_must_be_within_context
|
||||
errors.add(:irreversible, I18n.t('filters.errors.invalid_irreversible')) if irreversible? && !context.include?('home') && !context.include?('notifications')
|
||||
end
|
||||
end
|
||||
|
|
|
|||
34
app/models/custom_filter_keyword.rb
Normal file
34
app/models/custom_filter_keyword.rb
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# frozen_string_literal: true
|
||||
# == Schema Information
|
||||
#
|
||||
# Table name: custom_filter_keywords
|
||||
#
|
||||
# id :bigint not null, primary key
|
||||
# custom_filter_id :bigint not null
|
||||
# keyword :text default(""), not null
|
||||
# whole_word :boolean default(TRUE), not null
|
||||
# created_at :datetime not null
|
||||
# updated_at :datetime not null
|
||||
#
|
||||
|
||||
class CustomFilterKeyword < ApplicationRecord
|
||||
belongs_to :custom_filter
|
||||
|
||||
validates :keyword, presence: true
|
||||
|
||||
alias_attribute :phrase, :keyword
|
||||
|
||||
before_save :prepare_cache_invalidation!
|
||||
before_destroy :prepare_cache_invalidation!
|
||||
after_commit :invalidate_cache!
|
||||
|
||||
private
|
||||
|
||||
def prepare_cache_invalidation!
|
||||
custom_filter.prepare_cache_invalidation!
|
||||
end
|
||||
|
||||
def invalidate_cache!
|
||||
custom_filter.invalidate_cache!
|
||||
end
|
||||
end
|
||||
Loading…
Add table
Add a link
Reference in a new issue