Decline Search is broken

[Taluntain]

Good thing it's not the end of the week yet!

 

[DraQ]

Search is one of the most taxing things on the server, so... umm... not likely.

I'm sure hundreds of redundant threads and posts from lack of usable search will help lighten the load.

 

[Crooked Bee]

Good thing it's not the end of the week yet!

Almost there though.

 

[Nevill]

Good thing it's not the end of the week yet!

Shamelessly bugging again.

 

[Taluntain]

Oh yeah. Somebody really needs to do something about that, like later today.

 

[Taluntain]

ARE WE THERE YET?

 

[Nevill]

Well, it definitely looks much better now, thanks.

...

I don't suppose we could teach it to search for a full phrase instead of each keyword separately?

 

[Taluntain]

Tried using quotes?

 

[LeStryfe79]

No wonder everyone calls us "The Hell of the Internet".

 

[Nevill]

Tried using quotes?

I did, which is why I am asking.

Example. This thread is called 'Myth: A New Age CYOA'.

I am pretty sure the words "a new age" as a whole are only mentioned in the first two posts and maybe this one. Now try entering them in a search query, with proper quotes:
http://www.rpgcodex.net/forums/index.php?search/30791388/&q="A+New+Age"&t=post&o=date&c[thread]=89150

It offers 59 hits. I am not even talking about that it highlights the name of the thread in the search results. If the thread name was enough to make it to the list, every post in this thread would qualify, yet there are only 59 of them, so I assume it is just a visual quirk. It also highlights 'a' as if it qualifies for the query and regardless of where it is in the post, but does not highlight 'age'.

Spoiler

What I am talking about is that most hits have nothing to do with the query.

This post demonstrates best everything that is wrong with the search. It has neither 'new' nor 'age' in it, yet it still shows among the results. It has a word/letter 'a' in it, but almost every post has one, and it would result in more than 59 hits if that really was what the engine was looking for.

The only thing I understand about how the search algorithm works now is that it does not work as it should. Not sure if it is fixable and if the problem is even on your end, though.

Edit: forget it, it does not work, period. Search for 'CYOA' in that thread.

This post or this one displays as a hit. But there is no mention of 'CYOA' in it.

 

[DarkUnderlord]

Seems to be bringing it up because it's in the thread title.

 

[Taluntain]

That is weird, especially since there's no way to turn it off that I can see. I'll look into it.

 

[Nevill]

The thread has 6000+ posts. If that was the case I would have expected more hits.

Nevermind. Back to my original point.

Tried using quotes?

Sometimes it works and sometimes it does not. I can't figure out the pattern.

For example, searching the Playground for "Remind me when" gives 5 results, which seems to be correct:
http://www.rpgcodex.net/forums/index.php?search/30794703/&q="remind+me+when"&o=date&c[node]=38

Searching for 'Remind me when' (without quotes) gives the posts that have all three (?) words in them regardless of their relative positions:
http://www.rpgcodex.net/forums/index.php?search/30794756/&q=remind+me+when&o=date&c[node]=38

An example where the quotes do not work would be this thread. Try searching "the zhang" there and see what you get:
http://www.rpgcodex.net/forums/index.php?search/30794659/&q="the+zhang"&t=post&o=date&c[thread]=85971

It gives the same results you would get without the quotes:
http://www.rpgcodex.net/forums/index.php?search/30794845/&q=the+zhang&t=post&o=date&c[thread]=85971

The last case is unrelated to the thread title.

 

[Taluntain]

XF support speaks:

"Looking into it a bit more, the reason the results are being returned is because there are links in each post with the search terms in them.

This post for example: http://www.rpgcodex.net/forums/index.php?threads/search-is-broken.96950/page-2#post-3874525

Mentions that 'This post or this one displays as a hit. But there is no mention of 'CYOA' in it.'.

Each of the posts referenced there have linked URLs (hypertext) to threads with 'cyoa' in the URL:
http://www.rpgcodex.net/forums/index.php?threads/myth-a-new-age-cyoa.89150/page-126#post-3334422
http://www.rpgcodex.net/forums/index.php?threads/myth-a-new-age-cyoa.89150/page-230#post-3681017

Specifically 'The current leading options are:' and '1.D) Operation: Two Ways.'.

This is the expected behaviour as all of the post content is indexed, including hypertext."

(Yea, this is dumb since the text is only in the urls for SEO purposes and it's only throwing internal search results off, so I'll suggest that they add an option to omit urls within text from search. It's not possible currently.)

More:

"However, with regards to the highlighting, it's unrelated to the search process.
It's purely a visual thing and applied from left to right.
So as 'a' is matched first, 'age' won't be matched.

The letter 'a' is actually a stop word anyway, so nothing will ever be returned when searching for that.
It can be omitted from the search query and that will produce better results - e.g. "new age"."

For future reference - stop words and any words shorter than 3 characters are omitted from search, so don't bother entering them.

 

[Nevill]

I see.

So the word 'the' is a stop word, too, which explains this case:

Spoiler

An example where the quotes do not work would be this thread. Try searching "the zhang" there and see what you get:
http://www.rpgcodex.net/forums/index.php?search/30794659/&q="the+zhang"&t=post&o=date&c[thread]=85971

It gives the same results you would get without the quotes:
http://www.rpgcodex.net/forums/index.php?search/30794845/&q=the+zhang&t=post&o=date&c[thread]=85971

It is equivalent to merely searching for 'zhang'.

Is there anywhere I can find the list of stop words?

Edit: Wait! If any words shorter than 3 characters are omitted from the search, why would "Remind me when" return different results from "Remind when" (which returns no hits)? And if the stop words are treated normally when typed in quotes, why does 'the' gets dropped completely when searching for "the zhang"?

 

[tuluse]

Shouldn't strings inside quotes be treated as single "word" in this case anyways?

 

[Taluntain]

I see.

So the word 'the' is a stop word, too, which explains this case:

Spoiler

An example where the quotes do not work would be this thread. Try searching "the zhang" there and see what you get:
http://www.rpgcodex.net/forums/index.php?search/30794659/&q="the+zhang"&t=post&o=date&c[thread]=85971

It gives the same results you would get without the quotes:
http://www.rpgcodex.net/forums/index.php?search/30794845/&q=the+zhang&t=post&o=date&c[thread]=85971

It is equivalent to merely searching for 'zhang'.

Is there anywhere I can find the list of stop words?

Edit: Wait! If any words shorter than 3 characters are omitted from the search, why would "Remind me when" return different results from "Remind when" (which returns no hits)? And if the stop words are treated normally when typed in quotes, why does 'the' gets dropped completely when searching for "the zhang"?

The list:

"a", "an", "and", "are", "as", "at", "be", "but", "by",
"for", "if", "in", "into", "is", "it",
"no", "not", "of", "on", "or", "such",
"that", "the", "their", "then", "there", "these",
"they", "this", "to", "was", "will", "with"

We could get into this at length but we inevitably bump into the dilemma who the search primarily caters to - users searching for information in general or users searching for specific phrases. The tradeoff that works well in general for the first group (removal of stop words from searches to enhance performance and lower the overhead) will in some cases prevent the second group from immediately finding specific posts when they contain one or more stop words. This is where you have to weigh the pros and cons and decide whether the tradeoff is worth the benefit and in general, the consensus was that it is. However, looking at the more recent Elasticsearch docs on the subject, it appears that they've changed their minds to an extent. There have been a number of quite significant changes in how they handle searches and data in the more recent ES versions, so it'll be interesting to see how this works in the latest version.

Also: I've just found out that the 3-char limitation does not apply to ES at all. The setting is still there in the ACP, but it has no effect on the search when ES is enabed. So searches with characters shorter than 3 chars should work as well, as long as they're not on the list of stop words.

Shouldn't strings inside quotes be treated as single "word" in this case anyways?

Not when they contain a stop word.

 

[Infinitron]

Taluntain I think something's fucked.

 

[Taluntain]

Installing latest version of Elasticsearch, wait for the reindex to finish.

 

[Taluntain]

OK, should be working properly now - let me know if not.

 

[Nevill]

What do you know? It does treat strings in quotes as a single 'word' now, stop words or not!

 

[Nevill]

Is there a reason the conversations are not indexed in the search? Is it done to save resourses?

If not, can search be enabled across conversations as well?

 

[Taluntain]

XenForo doesn't came with an option for it at all so I assume it's down to resource usage again - indexing all the members' conversations would use up a lot of space on forums with large user bases, plus convos are something that is relatively rarely searched so I expect it was deemed unnecessary.

 

[valcik]

Couldn't be the shout box archives avoided by the search engine somehow, guys? When I'm trying to search for "Space Rangers" string, there's shitload of useless shout box crap listed in the results instead of valuable threads discussed by monocled codexians I'm looking for.

 

[Zep Zepo]

Couldn't be the shout box archives avoided by the search engine somehow, guys? When I'm trying to search for "Space Rangers" string, there's shitload of useless shout box crap listed in the results instead of valuable threads discussed by monocled codexians I'm looking for.

Have you tried putting http://www.rpgcodex.net/forums/index.php?members/shoutbox-archive-bot.18064/
on ignore?

Zep--