Recap

In Part 1, the merits and tradeoffs of subdirectories and subdomains were discussed.  The subdirectory strategy is typically superior to subdomains because subdomains suffer from keyword and backlink dilution.  The subdirectory strategy more effectively boosts a site's search rankings by ensuring that every keyword is attributed to the root domain instead of diluting across subdomains.

In the first part, our friend Bob set up a hosted Ghost blog at bobtopia.coolghosthost.com that he connected to blog.bobtopia.com using a CNAME DNS record.  But what if he wanted his blog to live at bobtopia.com/blog to gain the SEO advantages of subdirectories?

A reverse proxy like NGINX is normally needed to route traffic from subdirectories to remotely hosted services.  We'll demonstrate how to implement the subdirectory strategy with Cloudflare Workers and eliminate our dependency on NGINX. (Cloudflare Workers are serverless functions that run on the Cloudflare global network.)

Let's write a Worker that proxies traffic from a subdirectory – bobtopia.com/blog – to a remotely hosted platform – bobtopia.coolghosthost.com.  This means that if I go to bobtopia.com/blog, I should see the content of bobtopia.coolghosthost.com, but my browser should still think it's on bobtopia.com.

In the Workers editor, we'll start a new script with some basic configuration options.

// keep track of all our blog endpoints here
const myBlog = {
  hostname: "bobtopia.coolghosthost.com",
  targetSubdirectory: "/articles",
  assetsPathnames: ["/public/", "/assets/"]
}

The script will proxy traffic from myBlog.targetSubdirectory to Bob's hosted Ghost endpoint, myBlog.hostname.  We'll talk about myBlog.assetsPathnames a little later.

Requests are proxied from bobtopia.com/articles to bobtopia.coolghosthost.com (Uh oh... is because the hosted Ghost blog doesn't actually exist)

Next, we'll add a request handler:

async function handleRequest(request) {
  return fetch(request)
}

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

So far we're just passing requests through handleRequest unmodified.  Let's make it do something:

async function handleRequest(request) { 
  ...

  // if the request is for blog html, get it
  if (requestMatches(myBlog.targetSubdirectory)) {
    console.log("this is a request for a blog document", parsedUrl.pathname)
    const targetPath = formatPath(parsedUrl)
    
    return fetch(`https://${myBlog.hostname}/${targetPath}`)
  }

  ...
  
  console.log("this is a request to my root domain", parsedUrl.pathname)
  // if its not a request blog related stuff, do nothing
  return fetch(request)
}

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

In the above code, we added a conditional statement to handle traffic to myBlog.targetSubdirectory.  Note that we've omitted our helper functions here.  The relevant code lives inside the if block near the top of the function. The requestMatches helper checks if the incoming request contains targetSubdirectory.  If it does, a request is made to myBlog.hostname to fetch the HTML document which is returned to the browser.

When the browser parses the HTML, it makes additional asset requests required by the document (think images, stylesheets, and scripts).  We'll need another conditional statement to handle these kinds of requests.

// if its blog assets, get them
if ([myBlog.assetsPathnames].some(requestMatches)) {
    console.log("this is a request for blog assets", parsedUrl.pathname)
    const assetUrl = request.url.replace(parsedUrl.hostname, myBlog.hostname);

    return fetch(assetUrl)
  }

This similarly shaped block checks if the request matches any pathnames enumerated in myBlog.assetPathnames and fetches the assets required to fully render the page.  Assets happen to live in /public and /assets on a Ghost blog.  You'll be able to identify your assets directories when you fetch the HTML and see logs for scripts, images, and stylesheets.

Logs show the various scripts and stylesheets required by Ghost live in /assets and /public

The full script with helper functions included is:

// keep track of all our blog endpoints here
const myBlog = {
  hostname: "bobtopia.coolghosthost.com",
  targetSubdirectory: "/articles",
  assetsPathnames: ["/public/", "/assets/"]
}

async function handleRequest(request) {
  // returns an empty string or a path if one exists
  const formatPath = (url) => {
    const pruned = url.pathname.split("/").filter(part => part)
    return pruned && pruned.length > 1 ? `${pruned.join("/")}` : ""
  }
  
  const parsedUrl = new URL(request.url)
  const requestMatches = match => new RegExp(match).test(parsedUrl.pathname)
  
  // if its blog html, get it
  if (requestMatches(myBlog.targetSubdirectory)) {
    console.log("this is a request for a blog document", parsedUrl.pathname)
    const targetPath = formatPath(parsedUrl)
    
    return fetch(`https://${myBlog.hostname}/${targetPath}`)
  }
  
  // if its blog assets, get them
  if ([myBlog.assetsPathnames].some(requestMatches)) {
    console.log("this is a request for blog assets", parsedUrl.pathname)
    const assetUrl = request.url.replace(parsedUrl.hostname, myBlog.hostname);

    return fetch(assetUrl)
  }

  console.log("this is a request to my root domain", parsedUrl.host, parsedUrl.pathname);
  // if its not a request blog related stuff, do nothing
  return fetch(request)
}

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

There is one important caveat about the current implementation that bears mentioning. This script will not work if your hosted service assets are stored in a folder that shares a name with a route on your root domain.  For example, if you're serving assets from the root directory of your hosted service, any request made to the bobtopia.com home page will be masked by these asset requests, and the home page won't load.

The solution here involves modifying the blog assets block to handle asset requests without using paths.  I'll leave it to the reader to solve this, but a more general solution might involve changing myBlog.assetPathnames to myBlog.assetFileExtensions, which is a list of all asset file extensions (like .png and .css).  Then, the assets block would handle requests that contain assetFileExtensions instead of assetPathnames.

Bob is now enjoying the same SEO advantages as Alice after converting his subdomains to subdirectories using Cloudflare Workers.  Bobs of the world, rejoice!

Interested in deploying a Cloudflare Worker without setting up a domain on Cloudflare? We’re making it easier to get started building serverless applications with custom subdomains on workers.dev. If you’re already a Cloudflare customer, you can add Workers to your existing website here.

Reserve a workers.dev subdomain

Subdomain vs. Subdirectory: 2 Different SEO Strategies

Alice and Bob are budding blogger buddies who met up at a meetup and purchased some root domains to start writing.  Alice bought aliceblogs.com and Bob scooped up bobtopia.com.

Alice and Bob decided against WordPress because its what their parents use and purchased subscriptions to a popular cloud-based Ghost blogging platform instead.

Bob decides his blog should live at at blog.bobtopia.com – a subdomain of bobtopia.com. Alice keeps it old school and builds hers at aliceblogs.com/blog – a subdirectory of aliceblogs.com.

Subdomains and subdirectories are different strategies for instrumenting root domains with new features (think a blog or a storefront).  Alice and Bob chose their strategies on a whim, but which strategy is technically better?  The short answer is, it depends. But the long answer can actually improve your SEO.  In this article, we'll review the merits and tradeoffs of each. In Part 2, we'll show you how to convert subdomains to subdirectories using Cloudflare Workers.

Setting up subdirectories is trivial on basic websites.  A web server treats its subdirectories (aka subfolders) the same as regular old folders in a file system.  In other words, basic sites are already organized using subdirectories out of the box.  No set up or configuration is required.

In the old school site above, we'll assume the blog folder contains an index.html file. The web server renders blog/index.html when a user navigates to the oldschoolsite.com/blog subdirectory_._  But Alice and Bob's sites don't have a blog folder because their blogs are hosted remotely – so this approach won't work.

On the modern Internet, subdirectory setup is more complicated because the services that comprise a root domain are often hosted on machines scattered across the world.

Because DNS records only operate on the domain level, records like CNAME have no effect on a url like aliceblogs.com/blog – and because her blog is hosted remotely, Alice needs to install NGINX or another reverse proxy and write some configuration code that proxies traffic from aliceblogs.com/blog to her hosted blog. It takes time, patience, and experience to connect her domain to her hosted blog.

A location block in NGINX is necessary to proxy traffic from a subdirectory to a remote host

Bob's subdomain strategy is the easier approach with his remotely hosted blog.  A DNS CNAME record is often all that's required to connect Bob's blog to his subdomain.  No additional configuration is needed if he can remember to pay his monthly subscription.

Configuring a DNS record to point a hosted service at your blog subdomain

To recap, subdirectories are already built into simple sites that serve structured content from the same machine, but modern sites often rely on various remote services.  Subdomain set up is comparatively easy for sites that take advantage of various hosted cloud-based platforms.

Subdomains are neat. If you ask me, blog.bobtopia.com is more appealing than bobtopia.com/blog. But if we want to make an informed decision about the best strategy, where do we look?  If we're interested in SEO, we ought to consult the Google Bot.

Subdomains and subdirectories are equal in the eyes of the Google Bot, according to Google itself.  This means that Alice and Bob have the same chance at ranking in search results.  This is because Alice's root domain and Bob's subdomain build their own sets of keywords.  Relevant keywords help your audience find your site in a search. There is one important caveat to point out for Bob:

A subdomain is equal and distinct from a root domain.  This means that a subdomain's keywords are treated separately from the root domain.

What does this mean for Bob?  Let's imagine bobtopia.com is already a popular online platform for folks named Bob to seek kinship with other Bobs.  In this peculiar world, searches that rank for bobtopia.com wouldn't automatically rank for blog.bobtopia.com because each domain has its own separate keywords.  The lesson here is that keywords are diluted across subdomains.  Each additional subdomain decreases the likelihood that any particular domain ranks in a given search.  A high ranking subdomain does not imply your root domain ranks well.

In a search for "Cool Blog", bobtopia.com suffers from keyword dilution. It doesn't rank because its blog keyword is owned by blog.bobtopia.com.

Subdomains also suffer from backlink dilution.  A backlink is simply a hyperlink that points back to your site. Alice's attribution to a post on the etymology of Bob from blog.bobtopia.com does not help bobtopia.com because the subdomain is treated separate but equal from the root domain.  If Bob used subdirectories instead, Bob's blog posts would feed the authority of bobtopia.com and Bobs everywhere would rejoice.

The authority of blog.bobtopia.com is increased when Alice links to Bob's interesting blog post, but the authority of bobtopia.com is not affected.

Although search engines have improved at identifying subdomains and attributing keywords back to the root domain, they still have a long way to go.  A prudent marketer would avoid risk by assuming search engines will always be bad at cataloguing subdomains.

So when would you want to use subdomains?  A good use case is for companies who are interested in expanding into foreign markets.  Pretend bobtopia.com is an American company whose website is in English.  Their English keywords won't rank well in German searches – so they translate their site into German to begin building new keywords on deutsch.bobtopia.com. Erfolg!

Other use cases for subdomains include product stratification (think global brands with presence across many markets) and corporate internal tools (think productivity and organization tools that aren't user facing).  But unless you're a huge corporation or just finished your Series C round of funding, subdomaining your site into many silos is not helping your SEO.

If you're a startup or small business looking to optimize your SEO, consider subdirectories over subdomains.  Boosting the authority of your root domain should be a universal goal of any organization. The subdirectory strategy concentrates your keywords onto a single domain while the subdomain strategy spreads your keywords across multiple distinct domains. In a word, the subdirectory strategy results in better root domain authority. Higher domain authority leads to better search rankings which translates to more engagement.

Consider the multitude of disruptive PaaS startups with docs.disruptivepaas.com and blog.disruptivepaas.com.  Why not switch to disruptivepaas.com/docs and disruptivepaas.com/blog to boost the authority of your root domain with all those docs searches and StackOverflow backlinks?

Interested in switching your subdomains to subdirectories without a reverse proxy? In Part 2, we'll show you how using Cloudflare Workers.

Interested in deploying a Cloudflare Worker without setting up a domain on Cloudflare? We’re making it easier to get started building serverless applications with custom subdomains on workers.dev. If you’re already a Cloudflare customer, you can add Workers to your existing website here.

Reserve a workers.dev subdomain

I'll speak for myself — as someone who always learned just enough about DNS to get it working, then immediately forgot everything until the next time it broke. It was a vicious cycle until I convinced myself to learn it in depth.  Meanwhile, non-technical folks wisely avoid meddling in such dangerous affairs all together.

Surely, there must be a better way (this is a blog post after all).

Every day, thousands of Cloudflare users add DNS records to their Internet properties to configure awesome tools like G Suite, Shopify, Wordpress, Ghost, and thousands of others.  A new Cloudflare Apps feature allows apps to automatically set up and manage configurable DNS records on more than 12 million registered domains on the Cloudflare network. In short, Cloudflare Apps are here to alleviate the Internet’s collective DNS woes.

Gone are the days of tribulating over whether it’s A or CNAME you should set.  Gone are the days of puzzling between A and AAAA records while wondering what the heck happened to AA and AAA records?  Unload your DNS dysphoria onto highly trained developers experienced at explicating these burdensome questions today!

Did you know?

Are you a highly trained developer?  Cloudflare now provides the tools to build robust, powerful apps that automate DNS record management saving countless developer hours that are currently spent fiddling with DNS.  Instead of manually configuring records to integrate with your solution provider, build a Cloudflare App that does it for you and for everyone else on the Internet.

Pointless DNS is a demo app that showcases the new DNS feature by installing a (pointless) TXT record on any root or subdomain of your choice.  The TXT record is configured and managed by the Pointless DNS app.  Go ahead and install it to see automated DNS record management with Cloudflare Apps.

The TXT record name was set to "blog" during installation

After installation, head over to your DNS dashboard and you'll see the app doing its thing.  If you really want to, you can uninstall it from your Installed Apps page.

Pointless DNS automatically manages its TXT record on my blog subdomain

To start building your own DNS app, download create-cloudflare-app and open it in your preferred text editor.  Below, I’ll explain how Pointless DNS manages its configurable TXT record.

In the install.json file, you’ll find a dns field that looks like this:

# install.json

"dns": [{
    "type": "TXT",
    "content": "Managed TXT Record",
    "name": "Created by create-cloudflare-app",
    "ttl": 120
  }]

This says a TXT record will be set up and managed on the app installer's site, which can only be modified or deleted through the app’s configuration page.  Apps can create or modify any type of Cloudflare DNS record available in Cloudflare's DNS API.

Now let's add in some configuration. Add an install option and point it at your record — options.subdomain will be set to the installer's desire.

# install.json

{
  "options": {
    "properties": {
      "subdomain": {
        "order": 1,
        "type": "string",
        "title": "TXT Record Subdomain",
        "description": "The subdomain of your DNS record",
        "placeholder": "*Required - e.g. [your_domain].com",
        "required": true
      }
    }
  },
  "dns": [{
    "type": "TXT",
    "name": "{{options.subdomain}}",
    "content": "{{options.subdomain}} was set"
  }]
}

That's all there is to it.  Check out the Cloudflare Apps docs to learn more about about building DNS apps.  For additional inspiration, consider Mailchannels — a powerful email security solution that manages DNS records to fight off pesky spammers and phishermen.

Woof!

DNS records used to be hard, but a new breed of Cloudflare Apps makes them easy.  Cloudflare Apps-based automated DNS configuration gives developers the opportunity to be the hero people want and the hero they need.

NOTE: This post was updated 5 hours after initial publication to include additional, relevant details.