[static filter syntax] irregularities when no wildcard is used #1065

Activity

1. gorhill commented on Dec 14, 2015

   

   gorhill

   on Dec 14, 2015

   Owner

   More actions

   in uBO it seems you have to add at least

   By design. Unlike ABP, uBO does not use RegExp by default internally for most filters, only when it can not be avoided -- for efficiency reason. An asterisk is required to explicitly declare that any character segment is allowed in one spot, this includes the beginning/end of a filter, otherwise it is assume the beginning/end is a word boundary (if defining word as a sequence of any of the character from the set %0-9A-Za-z).

   👍React with 👍1Reacted by uBlock-user
2. gorhill

   closed this as completedon Dec 14, 2015
3. lewisje commented on Dec 15, 2015

   

   lewisje

   on Dec 15, 2015

   More actions

   Could you put that into the "Static filter syntax" section of the wiki? I too was under the impression that blocking rules had implicit wildcards at beginning and end (with the exception that if the rule would otherwise start and end with a forward slash /, an asterisk * needed to be prepended or appended, usually the latter, to keep it from being interpreted as a regex).

   This actually makes uBO more efficient than ABP and the like, because it's possible to specify that a filter begins and ends on a word boundary without using a regex.
4. gorhill commented on Dec 15, 2015

   

   gorhill

   on Dec 15, 2015

   Owner

   More actions

   I too was under the impression that blocking rules had implicit wildcards at beginning and end

   They have implicit wildcards, except after implying word boundaries.

   This actually makes uBO more efficient than ABP

   The kind of filters as seen in OP are quite rare in EasyList/EasyPrivacy. They leads to inefficient filtering, because they are broad, can't be tokenized (if following ABP rules), and with such filter every single URL must be checked against every single one of these untokenizable filters. Given their rarity in EasyList/EasyPrivacy, this is not really the explanation of why uBO's static network filtering engine is more efficient.

   For other cases where the start or end is a word character, it's clear the intended function was not to have a wildcard interpretation. For example, in EasyList:

   120x500.gif|
   

   I really doubt that the purpose of this filter is to block all GIF images which URL ends with 120x500.gif, but rather all GIF images which size is 120 x 500, not 5120 x 500, or 10120 x 500. (correction: ok, I realize the filter could be to match zebra120x500.gif and lion120x500.gif -- I may chose to remove the word boundary assumption.. reopening).

   Implicit word boundaries is what is pretty much always happening anyways in EasyList/EasyPrivacy. I am sure that nowadays, for performance consideration re. ABP, official filter list maintainers avoid filters like in OP.

   Regarding the evaluation of network static filters, the upperhand of uBO versus ABP is not because of the differing interpretation re. leading/tailing wildcards, but rather because majority of filters are evaluated not using a regex in uBO (this would still be true if implying wildcards like ABP), while all filters are translated into regexes in ABP (someone correct me if I am wrong).

   Typically regex will cause the whole URL to be scanned. With uBO, the token extracted from the URL is used to first lookup a bucket of potentially matching filters, and key to the performance of static network filter evaluation, the position of the token is used as an anchor to perform a plain string comparison, no need to scan the whole URL:

   filter:  /top728x90.
   tokens:   top728x90
   offset:   1
   
   URL:     https://www.example.com/images/top728x90.png?t=145009383
   tokens:  https   www example com images top728x90 png   145009383
   offset:  0       8   12      20  24     31        40    46
   

   So uBO matches the anchor of the filter token to the anchor of the URL token:

   tokens:                                 top728x90
   offset:                                 1
   tokens:  https   www example com images top728x90 png   145009383
   offset:  0       8   12      20  24     31        40    46
   

   Segment of string to compare in URL to filter string is at position (31 - 1 = 30):

   filter:                                /top728x90.
   URL:     https://www.example.com/images/top728x90.png?t=145009383
   offset:                                30
   

   In the end it's a mere string comparison: URL.startsWith('/top728x90', 30).

   There are other different optimizations too for static network filtering evaluation, like using the request type/party as hash key to narrow the number of filters to evaluate; or another optimization which is related to the compact storing of plain hostname-based filters -- which is the most common occurrence in uBO.
5. gorhill

   reopened this on Dec 15, 2015
6. gorhill

   closed this as completedon Dec 15, 2015
7. gorhill commented on Dec 15, 2015

   

   gorhill

   on Dec 15, 2015

   Owner

   More actions

   I prefer to leave it as is, it's just makes more sense to explicitly use a wildcard if we want to cancel word boundaries (i.e. *120x500.gif using above example).
8. gorhill commented on Feb 11, 2016

   

   gorhill

   on Feb 11, 2016

   Owner

   More actions

   Here is a real case where implying a wildcard causes an issue: http://www.858.photos/.

   The images in the carousel at the top of the page are prevented from loading, because their URL is something like http://www.858.photos/wp-content/uploads/2015/10/Blue-Hour-Exterior-18512-HDR-1160x600.jpg, while there is the filter 160x600.jpg| in EasyList.
10. gorhill commented on Mar 16, 2016

    

    gorhill

    on Mar 16, 2016

    Owner

    More actions

    Yes, /adblockpopunder is interpreted differently in uBO compared to ABP. I believe ABP should still change this to adopt the word-boundary interpretation, it's way more sensible -- no word boundary is rather the exception in majority of cases, and when no word boundary is required, the asterisk can be used to specify this (/adblockpopunder*). Currently, if someone really wants to force a word boundary, there is nothing in ABP to specify this.