🐛 Fixed /p/ redirects not being indexed by search engines (#19864)

ref
[ENG-741](https://linear.app/tryghost/issue/ENG-741/🐛-our-robotstxt-config-causes-indexing-issues-for-customers-who-have)

`/p/` has been dropped from the `robots.txt` file so that search engines
can index the pages at these locations. In the event that the page at
the location is a preview page, the existing robots meta tag on the page
will prevent indexing.

ghost/core/core/frontend/public/robots.txt

@@ -1,7 +1,6 @@
 User-agent: *
 Sitemap: {{blog-url}}/sitemap.xml
 Disallow: /ghost/
-Disallow: /p/
 Disallow: /email/
 Disallow: /r/
 Disallow: /webmentions/receive/

ghost/core/test/e2e-frontend/default_routes.test.js

@@ -319,12 +319,9 @@ describe('Default Frontend routing', function () {
                 .expect(200)
                 .expect(assertCorrectFrontendHeaders);
 
-            // The response here is a publicly documented format users rely on
-            // In case it's changed remember to update the docs at https://ghost.org/help/modifying-robots-txt/
             res.text.should.equal(
                 'User-agent: *\n' +
                 'Sitemap: http://127.0.0.1:2369/sitemap.xml\nDisallow: /ghost/\n' +
-                'Disallow: /p/\n' +
                 'Disallow: /email/\n' +
                 'Disallow: /r/\n' +
                 'Disallow: /webmentions/receive/\n'