Search engines "ghosting" these forums??

[Techwright]

I've a confession to begin with:  I often get weirdly lazy and if I've several browser tabs open, rather than also opening the game launcher just to link to the forums, I'll just type "Coh homecoming forums" at the top of another, and DuckDuckGo will pull up a list starting with the link to the forums.  Our unofficial wiki is almost always next.  Other things like the Reddit appear farther down the list.

 

At least, that's the way it was until at some point in the past 24 hours, which is when I last attempted this method.  Now when I attempt it, using the same phrasing, the forums are not listed.  The wiki appears first, usually followed by a Reddit link, and a host of other things that are not the link to the forums.  DDG asks if perhaps I wanted "City of Heroes Homecoming Forums".   Odd, the engine never did that before, but okay, yes, sure.  I select what it suggest and... no forums.  Again the wiki is the lead, Reddit's also a top piece, and a host of other things.  Several attempts later and I cannot get DuckDuckGo to reveal the link.  As part of my experimentation, I drilled down to subsequent pages and did not find a direct link.  I also found it curious that the engine mysteriously ran out of links a few pages into the search.

 

So, donning my tin foil hat, I begin wondering if there's a conspiracy behind this.  I make the same attempt on Google search.  I use my standard search phrase.  The forums link appears immediately and at the top of the list.  I don't like Google Search, but in this case, bravo, good for them.  I then attempt other search engines.  So far, I've found the forums no longer appear in the listing for:

 

• Duck Duck Go
• Bing  - Bing puts the Homecoming Reddit at top of its searches, with the wiki second
• Yahoo Search  -- Yahoo likewise leads with the Reddit, seconds with the wiki.  However, it, apparently inadvertently, reveals the Homecoming forums link in a side box to the right of the search results column.  This side box is a link to the wikipedia article talking about Homecoming Servers, and as part of that Wikipedia arrangement, lists the link Wikipedia points to:  forums.homecomingservers.com

 

Now my tin foil hat is feeling electrified.  What is going on here?  Did the forums do something search engines consider bad?  Did some AI feel we slighted it?  Did we not bribe the proper people?  Are the malcontent with Homecoming's success working behind the scenes to hurt it somehow?  Or...?

 

Yeah, for me and all other current players, it is a simple matter of opening the game launcher and clicking on "Forums".  I'm thinking more towards long-term effects for others who, like me, at some point learn the game is functioning again and go searching for links.  And yes, there are ways around this.  Going into the wiki, will reveal a forums link, for example.  But the clean, direct way has apparently been remove from at least 3 of the big name search engines and I find this at least odd, if not a bit disturbing.

 

 

 

(Now where did I put that phone number for The Question? This is right up his alley.)

[Luminara]

Verified what you're seeing on DuckDuckGo, these forums don't show up at all in any searches, with any of the keyword combinations I tried, with Safe Search and AI features disabled, or anything else.  None of the results bring the forums up on the list, not even suggested searches or related searches work.  It's possible that the anti-spam/bot security measures aren't interacting happily with those three search engines.

 

You couldn't pay me to try other two, though, so that's as far as my investigation is going.

[Lunar Ronin]

DuckDuckGo and Yahoo! are essentially wrappers for Bing, so it's not at all surprising that they would show the same search results as they all use the Bing search engine.  As for why Bing has deprioritized the Homecoming Forums, no idea.  I checked the robots.txt file to see if Homecoming is blocking Bing for some reason, but nothing there.  But Bing is a pretty bad search engine anyhoo.

 

There are actual search engines: Bing, Brave Search, Google, Kagi, Mojeek, and Yandex.  Others are either a wrapper for Bing, or are metasearch engines (meaning they search multiple search engines for you simultaneously).  FWIW, I personally use and recommend Kagi, although it isn't free.  Brave Search and Mojeek are my secondary recommendations.  Bing and Google are just bad, especially since the LLM wars started.

[Luminara]

2 minutes ago, Lunar Ronin said:

DuckDuckGo and Yahoo! are essentially wrappers for Bing, so it's not at all surprising that they would show the same search results as they all use the Bing search engine.

 

[Lunar Ronin]

2 minutes ago, Luminara said:

 

 

Agreed.  If you want actual search results, try Brave Search, Kagi, and/or Mojeek.

[Techwright]

14 hours ago, Lunar Ronin said:

 

There are actual search engines: Bing, Brave Search, Google, Kagi, Mojeek, and Yandex. 

I'm reasonably sure those last three were names of Ravagers in the MCU Guardians of the Galaxy films. LOL.

 

Duck Duck Go is a wrapper for Bing?  I'm...disappointed.  The implications of its advertising seem to point to a true autonomy, but sadly...

 

14 hours ago, Luminara said:

Verified what you're seeing on DuckDuckGo, these forums don't show up at all in any searches, with any of the keyword combinations I tried, with Safe Search and AI features disabled, or anything else.  None of the results bring the forums up on the list, not even suggested searches or related searches work.  It's possible that the anti-spam/bot security measures aren't interacting happily with those three search engines.

 

Thanks for confirming I'm not crazy, at least in this matter..  I do wonder how we offended Bing.

 

I'm not in a position to pay Kagi, but I did my standard search on Brave Search, Mojeek, and Yandex.  Results were spot on for all three, so thanks for the tip @Lunar Ronin.  Are there any popular other "wraps" for Bing besides Duck Duck Go and Yahoo!?

[Lunar Ronin]

1 hour ago, Techwright said:

I'm not in a position to pay Kagi, but I did my standard search on Brave Search, Mojeek, and Yandex.  Results were spot on for all three, so thanks for the tip @Lunar Ronin.  Are there any popular other "wraps" for Bing besides Duck Duck Go and Yahoo!?

 

Ask.com and Startpage are wrappers for Google.  In addition to DuckDuckGo and Yahoo!, Ecosia (an up and coming "environmentally friendly" search engine), Excite, HotBot, Lycos, Qwant (another popular privacy-friendly alternative to DuckDuckGo), and You.com all are wrappers for Bing.

 

There are also up and coming search "AIs," namely SearchGPT and Perplexity.  But note that quite a few websites are at least attempting to block all known LLM crawlers, including the websites that I own.

Edited August 11 by Lunar Ronin

[Cipher]

Hey,

 

Thanks for bringing this up for us. I've confirmed that Bing still has our websites indexed, but they are no longer showing in search results. I've reached out to Bing Webmaster Support to look into this issue and will provide an update when one is available.

[honoroit]

Searching for 'city of data' finds the forums, useful every time I forget its url.

 

Thanks for seeing about search boosts, it's important my stuff can be found, because I'm famous.

 

-- figs, the famous

[Techwright]

On 8/11/2024 at 5:48 PM, Cipher said:

Hey,

 

Thanks for bringing this up for us. I've confirmed that Bing still has our websites indexed, but they are no longer showing in search results. I've reached out to Bing Webmaster Support to look into this issue and will provide an update when one is available.

 

15 days after, and I just ran a test. Bing, Duck Duck Go, and Yahoo still have the same results.  I realize that Microsoft might be moving at a snails pace for whatever reason.

[Cipher]

On 8/27/2024 at 2:35 AM, Techwright said:

 

15 days after, and I just ran a test. Bing, Duck Duck Go, and Yahoo still have the same results.  I realize that Microsoft might be moving at a snails pace for whatever reason.

This one is out of our hands unfortunately. Bing Webmaster support gave us the typical "Thank you for your patience during our investigation. After further review, it appears that your site had issues with meeting the standards set by Bing to remain indexed the last time it was crawled. To ensure that this was not a false flag, I also escalated the issue to the next level, and they manually reviewed your site and confirmed that it is in violation of our webmaster guidelines detailed here: Bing Webmaster Guidelines. We are not able to provide specifics for these types of issues because the relevant teams do not share reports with us. We recommend that you review our Webmaster Guidelines, especially the section Things to Avoid, and thoroughly check your site for any deliberately or accidentally employed SEO techniques that may have adversely affected your standing in Bing and Bing-powered search results. We will not be able to add your site to the index while Bingbot is finding it in violation of Webmaster Guidelines. We have no control over this process, and you will need to make quality changes to your site and wait for the Bingbot's automated crawling process to agree that the site passes the webmaster standards before it can be served."

 

When looking over the site in the webmaster tools, it shows that our sites are indexed and that there are no errors detected by Bingbot which would prevent indexing and appearing in search results. Unfortunately that means that the cause of this "ban" is likely external to our sites - things like having backlinks from sketchy websites, a surge in social media presence, etc. can all cause these sorts of bans. The only theory I have at the moment is that there is an AI assistant website showing with over 10,000 backlinks to our website in the webmaster tools which far exceeds anything else, and that may be tripping some flag.

[Luminara]

1 hour ago, Cipher said:

The only theory I have at the moment is that there is an AI assistant website showing with over 10,000 backlinks to our website in the webmaster tools which far exceeds anything else, and that may be tripping some flag.

 

[Techwright]

3 hours ago, Cipher said:

This one is out of our hands unfortunately. Bing Webmaster support gave us the typical "Thank you for your patience during our investigation. After further review, it appears that your site had issues with meeting the standards set by Bing to remain indexed the last time it was crawled. To ensure that this was not a false flag, I also escalated the issue to the next level, and they manually reviewed your site and confirmed that it is in violation of our webmaster guidelines detailed here: Bing Webmaster Guidelines. We are not able to provide specifics for these types of issues because the relevant teams do not share reports with us. We recommend that you review our Webmaster Guidelines, especially the section Things to Avoid, and thoroughly check your site for any deliberately or accidentally employed SEO techniques that may have adversely affected your standing in Bing and Bing-powered search results. We will not be able to add your site to the index while Bingbot is finding it in violation of Webmaster Guidelines. We have no control over this process, and you will need to make quality changes to your site and wait for the Bingbot's automated crawling process to agree that the site passes the webmaster standards before it can be served."

 

When looking over the site in the webmaster tools, it shows that our sites are indexed and that there are no errors detected by Bingbot which would prevent indexing and appearing in search results. Unfortunately that means that the cause of this "ban" is likely external to our sites - things like having backlinks from sketchy websites, a surge in social media presence, etc. can all cause these sorts of bans. The only theory I have at the moment is that there is an AI assistant website showing with over 10,000 backlinks to our website in the webmaster tools which far exceeds anything else, and that may be tripping some flag.

 

Um...wow.  It's interesting that other search engines don't have a beef with us.  Okay, thanks.  I'm sorry it is this way, but I'm deeply appreciative of the effort made, @Cipher.

 

 

Edited August 31 by Techwright

[MistressOhm]

Quote

AI assistant website

 

Sounds like that one needs to be put into the "Go away, bot" blackhole.  Perhaps Bing/Microsoft is jealous that another LLM is trawling and encroaching on their monopoly?

[catsi563]

just saw this did a quick check and can confirm not showing up on bing/edge

 

googles got us ok

[brass_eagle]

I've noticed this for quite some time, just keeping the issue current. For me, I was able to find City of Heroes Homecoming last (by memory).

 

Since this is out of our hands, is there any way to convince whoever runs the Homecoming Wiki to provide a link in plain sight? I use the "Getting Started" link but that's a scroll. And every time I go there I get a pop-up to "update my bookmark" when I'm really trying to get here.

[Cipher]

Unfortunately Bing is still blocking our main website from appearing in search results per the screenshot below.

 

Their tooling is unable to identify any issues when testing against the live site and states that it is able to be indexed again, so we've requested it to be indexed. Given that their support team was unable to disclose the reasons we were pulled out of search results in the first place though, and that we could not identify any violations of their webmaster guidelines at the time, I'm not holding my breath.

 

Index result:

 

Live result: