• Login
  • Register

DNN Forums

A community discussion page. We're starting from scratch, so...let's get the party started!

bot detection

You are not authorized to post a reply.
Sort:


Basic Member


Posts:94
Basic Member

     I am not an expert here but I can see a reason to possibly want to see those exceptions.  Would there be a way to detect if it's a bot.  Also if you can detect a bot maybe we can also control the bots we want to allow.   When a bot is detected add it to a list that an admin can allow/deny it's presence.  I have wanted to think of a solution for this purpose.  This exception gives a clue to a possibility.

    There are a lot of rogue bots that probably dont care about a robot.txt file. 

    https://github.com/dnnsoftware/Dnn.Platform/issues/3380

    This issue shows there can be a lot of bot activity that can lead to huge log files due to the antiforgery key missing from the requests. 

    Would it be practicle to put an IF this error happens to send the bot type information to a collection in a database and allow a host to allow/deny it's ability to interact with the website.

    I get a lot of times ones that go to pages that dont exist such as for a wordpress admin login panel.  This should be a trap that auto-bans any further requests from that visitor.  I had thought of how to enhance the quality of traffic being allowed on the DNN Portals and thought maybe we could come up with some type of solution.

    I do not see an area to chat about the platform itself yet so I am just posting these types of discussions here in the Platform API which I am sure will upset a few readers... move them if necessary please to the right spot.

    Any thoughts on this issue here as it would be nice to get all the bad page request errors and antiforgery bot errors to be a minimal impact on the site logs.  Being a past game mod developer I know the challenges when dealing with bots.  It is like adding a captcha to the site and verification email, you just have to do it or the bots overrun your site.



    Basic Member


    Posts:95
    Basic Member

      It has little to do with rogue bots. The bots that hit those pages get a page returned without an anti-forgery cookie and that is perfectly fine. It looks like DNN is actually returning the correct error because the cookie is indeed missing but we need to find a way to suppress those errors.
      DNN Technology Team | Area Lead Testing | Read the manual https://dnndocs.com/index.html


      Basic Member


      Posts:94
      Basic Member

        I understand the issue as I have been dealing with it for a very long time. In fact this is causing more issues that are related such as having to purge site logs due to it growing out of control.

        The two most common errors have to do with malicious bots hitting pages that do not exist and the antiforgery  non-malicious bots from google bot indexing your site. There are "good" bots and "bad" bots.  Each administrator may have a different opinion of a "Good bot" as well.

        I want to use this as a way to tackle the "bad" bots. There are things you will notice in common as far as the bad bots go that are going to separate them from the "good" bots. I believe using the info from the browser type is going to tell you maybe if it is good and give a small clue along with IP addresses such as a geo filter.

        What needs to happen in my opinion what I would love to see done is creating a way to detect the Good bots and allow them to browse the site without errors in the logs. I believe possibly in the error log or another area in the persona bar there could be an option to views these specific errors and allow an admin to "Add to Safe List" similar to junk mail filter in mail programs. It is the only way I believe... Known registered IP's of safe bots can be added and maintained as a list. And possibly also known bad bots but that list is toooooooooo huge so forget it. Lets focus on the far less number of good bots.

        The anti-forgery issues can then also help detect "good" or bad and handle them according to this "safe list".  The unsafe it can then simply reject those connections to the site and maybe put one message per ip that this bot was blocked X many times with maybe the last 5 or 10 recorded...

        Maybe this makes sense but I think we can kill two birds with one stone here...

        A temporary fix would be something more like what is discussed on github for a solution to get by if desired to set it up like this.  Put an option to supress that message... or any message for that matter incase it is a known issue and you dont need to be notified more about it. But that is up to an admin as others have pointed out it is good to know what is going on with your site with detials such as what this error provides.

        Does the bot still load the page?  Will it index material?  This could even have bigger issue if it stops bot from indexing your site as well.  Not saying it does as I a would not doubt the bot gets around it.



        Basic Member


        Posts:94
        Basic Member

          So interesting enough I found release mode in Visual Studio. I knew there was a step missing I kept building not realizing I was in debug and I would not get a package built to test my work for a few months now. I feel like such an old newbie it has been a long time since I have used Visual Studio but I should have remembered this I think it is funny :)

          I will play with some ideas I have cooking in my head maybe I can find a way to handle this in a fun way :)

          I am still working on my IDE setup... I need to build a seperate machine and possibly a laptop I am debating which way to go here maybe some advice here or suggestions before I build an awesome setup.    I will throw a thread up for the fun of it see what you all are geared up with.
           



          Basic Member


          Posts:51
          Basic Member

            On thing you can do is put the site behind a service like Cloudflare. One of the things you can use it for is to block bots and other "bad browsers" before they ever reach your site. It's got a free tier, so it's worth looking into.
            You are not authorized to post a reply.

            These Forums are dedicated to discussion of DNN Platform.

            For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

            1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
            2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
            3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
            4. No Flaming or Trolling.
            5. No Profanity, Racism, or Prejudice.
            6. Site Moderators have the final word on approving / removing a thread or post or comment.
            7. English language posting only, please.

            Would you like to help us?

            Awesome! Simply post in the forums using the link below and we'll get you started.

            Get Involved