• Login
  • Register

DNN Forums

A community discussion page. We're starting from scratch, so...let's get the party started!

what is the value of the current content in robots.txt

You are not authorized to post a reply.
Sort:


New Member


Posts:14
New Member

    Hi,

    I got a question from a client about the security information the robots.txt file is exposing. It now shows a list of directories. Wouldn't it be better if these directories wouldn't be exposed or shown?

    My client (based on some pen test advise) now considers whitelisting in the robots.txt. I think that does not make sense

    I like to hear your opinions

     

    Currently it contains:

    # Begin robots.txt file
    #/-----------------------------------------------\
    #| In single portal/domain situations, uncomment the sitmap line and enter domain name
    #\-----------------------------------------------/
    #Sitemap: http://www.DomainNamehere.com/sitemap.aspx
    
    User-agent: *
    Disallow: /*/ctl/		# Googlebot permits *
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /Install/
    Disallow: /Providers/
    Disallow: /Activity-Feed/userId/	# Do not index user profiles
    
    # End of robots.txt file

     



    Veteran Member


    Posts:484
    Veteran Member

      Peter,

      robots.txt is originally designed as a blocklist (as the word "blacklist" is not politically correct and racist nowadays), therefore I think you're right. It could block too many "good" robots if you only allow some. And "bad" robots give a sh... about your robots.txt file.

      Happy DNNing!
      Michael

      Michael Tobisch
      DNN★MVP

      dnnWerk Austria
      DNN Connect


      New Member


      Posts:14
      New Member

        thanks Michael, maybe I add an issue and a pull request, see what can of worms that opens...
        You are not authorized to post a reply.

        These Forums are dedicated to discussion of DNN Platform.

        For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

        1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
        2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
        3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
        4. No Flaming or Trolling.
        5. No Profanity, Racism, or Prejudice.
        6. Site Moderators have the final word on approving / removing a thread or post or comment.
        7. English language posting only, please.

        Would you like to help us?

        Awesome! Simply post in the forums using the link below and we'll get you started.

        Get Involved