AFFiNE/packages/frontend/core/public/robots.txt

# If you would like to crawl AFFiNE public workspace contact us via developer@toeverything.info

User-agent: *
Allow: /
Disallow: /oauth/*
Disallow: /workspace/*
Disallow: /public-workspace/*
Disallow: /auth/*
Disallow: /sign-in
Disallow: /_next/*
Disallow: /invite/*
Disallow: /*/invite/*
Disallow: /api/*

User-agent: PetalBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Barkrowler
Disallow: /
feat: update crawler rules (#6369) 2024-03-29 04:33:12 +03:00			`# If you would like to crawl AFFiNE public workspace contact us via developer@toeverything.info`

			`User-agent: *`
			`Allow: /`
			`Disallow: /oauth/*`
			`Disallow: /workspace/*`
			`Disallow: /public-workspace/*`
			`Disallow: /auth/*`
fix(core): oauth popup blocked in safari (#8144) fix(core): oauth popup blocked in safari fix(core): standarize auth routes fix AF-1352 2024-09-06 11:26:27 +03:00			`Disallow: /sign-in`
feat: update crawler rules (#6369) 2024-03-29 04:33:12 +03:00			`Disallow: /_next/*`
			`Disallow: /invite/*`
			`Disallow: //invite/`
			`Disallow: /api/*`

			`User-agent: PetalBot`
			`Disallow: /`

			`User-agent: BLEXBot`
			`Disallow: /`

			`User-agent: Bytespider`
			`Disallow: /`

			`User-agent: Barkrowler`
			`Disallow: /`