How Googlebot Crawls Websites: Understanding Robots.txt, Sitemaps, and SEO Flow

Introduction

Have you ever wondered how your website shows up on Google? It all starts with a process called crawling, where Google’s robot (called Googlebot) visits your website, reads your content, and decides where it should appear in search results.

In this blog, we’ll explore how Googlebot works, what the robots.txt file and sitemap.xml do, and how all of this connects to SEO and getting your site to the first page of Google.

What is Googlebot?

Googlebot is a special program (or bot) created by Google. Its job is to “crawl” the web — visiting websites, reading pages, and collecting data to store in Google’s database.

It works like this:

• It finds new pages by following links.
• It checks old pages for updates.
• It sends this data back to Google, which then decides how to rank your pages in search results.

What is Robots.txt?

The robots.txt file is a simple text file placed in the root folder of your website. It tells search engine bots which pages they are allowed or not allowed to visit.

Example of a robots.txt file:

• User-agent: * means all bots (Google, Bing, etc.)
• Disallow: /admin/ means bots should not visit this folder
• Allow: / means bots can crawl the entire website

This file is useful if you want to hide private or sensitive pages from search engines.

What is Sitemap.xml?

A sitemap is an XML file that lists all the important pages on your website. It helps search engines find and understand your website structure faster. A sitemap might look like this:

You can submit your sitemap to Google through Google Search Console. This helps make sure all your pages are found, especially new or deep-linked pages that aren’t easily accessible.

How Googlebot Crawls and Indexes Your Site – The Flow

Here’s a simple flow of how it works:

1. Googlebot checks your robots.txt file.
2. It follows allowed links and starts crawling your pages.
3. It reads your sitemap.xml (if provided) to discover more pages.
4. Content from your pages is stored in Google’s index.
5. Based on quality, keywords, and links, Google ranks your pages in search results.



How SEO Helps You Rank on Google

Crawling and indexing are only part of the process. The next step is ranking, which depends on SEO (Search Engine Optimization).

Here are the main parts of SEO:

1. On-Page SEO
• Use proper keywords in titles, headings, and content.
• Add meta descriptions.
• Use clean URLs, fast loading pages, and mobile-friendly design.
2. Off-Page SEO
• Get backlinks (other websites linking to you).
• Share content on social media.
• Build trust and authority.
3. Technical SEO
• Use a valid robots.txt file.
• Keep your sitemap updated.
• Fix broken links and duplicate content.
• Use schema markup (structured data).



Common Mistakes to Avoid

• Accidentally blocking Googlebot in robots.txt.
• Forgetting to submit a sitemap.
• Slow page speed or not mobile-friendly.
• Using duplicate or thin content.
• Ignoring crawl errors in Google Search Console.



Best Practices for SEO & Googlebot Crawling

• Create and submit a sitemap regularly.
• Always check your robots.txt file after changes.
• Use internal linking wisely.
• Monitor your site’s health in Google Search Console.
• Focus on quality content that helps users.



Conclusion

Understanding how Googlebot crawls your website, and how robots.txt and sitemap.xml work, gives you better control over how your content appears on Google. Combine this with solid SEO, and your website will be on the right path to reach the top of search results.

If you have any questions you can reach out our SharePoint Consulting team here.

PDF Text Not Displaying on macOS (Adobe Acrobat Pro)? Font Compatibility & Fix

Issue Summary:

When generating PDFs and opening them via Adobe Acrobat Pro on macOS, certain text content was not displaying properly. This issue was traced back to unsupported or improperly embedded web fonts, particularly WOFF/WOFF2, which are not supported for embedding in PDFs on macOS.

Key Findings & Root Cause:

• macOS does not embed WOFF/WOFF2 in PDFs.

• Adobe Acrobat Pro has inconsistent rendering with web fonts.

• Some fonts like "Exo 2" from Google Fonts fail to render unless properly converted and embedded.

macOS-Compatible Font Formats:

Font Format	Extension	Supported	Embeds in PDF
TrueType	.ttf	✅	✅
OpenType	.otf	✅	✅
AAT / DFTT	.dfont	✅	✅
WOFF/WOFF2	.woff / .woff2	⚠️ Web only	❌

System Fonts Pre-Installed in macOS:

• Sans-serif: Helvetica, Arial, SF Pro

• Serif: Times New Roman, Georgia, Palatino

• Monospace: Courier New, Menlo

Solutions & Best Practices:

1. Use widely supported fonts: Arial, Verdana, Helvetica, SF Pro.

2. Convert WOFF/WOFF2 fonts to .ttf or .otf formats before PDF generation.

3. Use @font-face with local font files:

    @font-face {  
     font-family: 'Exo 2';  
     src: url('/fonts/Exo2-Regular.ttf') format('truetype');  
    }  
   

4. In Puppeteer: Set embedFonts: true, ensure fonts are loaded before page.pdf() is called.

5. For critical compatibility, prefer Arial or Verdana in TrueType format.

Resolution Achieved:

• Switching to Verdana in TrueType format (.ttf) and ensuring it was properly embedded in the PDF resolved the issue across macOS devices and Adobe Acrobat Pro.

• Paging layout issues with Verdana were noted but considered minor and fixable.

Recommendation:

Stick to TrueType system fonts for maximum compatibility in PDF generation for macOS users. Avoid relying on WOFF/WOFF2 or Type 1 fonts, and always test in native apps like Adobe Acrobat Pro post-generation.

If you have any questions you can reach out our SharePoint Consulting team here.

A Comprehensive Guide to Microsoft Planner: Features, Benefits, Challenges, and Tips for Effective Use

Microsoft Planner is one of the task management tools included with Microsoft 365. It enables teams to create, assign, and organize tasks while fostering collaboration and providing clear visibility into project progress. With a visual, user-friendly interface, Planner offers a kanban-style experience similar to Trello or Asana but stands out for its deep integration with other Microsoft 365 apps like Teams, Outlook, and SharePoint. 

What is Microsoft Planner? 

Microsoft Planner is a visual task management application that tracks and arranges work using a kanban-style board. It allows users to: 

• Create task boards (called plans), 
  
• Organize tasks into buckets (categories), 
  
• Assign tasks to team members, 
  
• Set due dates and task priorities, 
  
• Monitor progress through status updates (Not Started, In Progress, Completed). 

While it's not a full-scale project management solution like Microsoft Project, Planner is ideal for lightweight, team-based task coordination. It’s seamlessly integrated with Microsoft 365 Groups, making it an excellent fit for teams already working within Outlook, Teams, or SharePoint. 

Why & When to Use Microsoft Planner 

Why use it: 

• To visually organize team tasks and projects. 
  
• To assign responsibilities and monitor accountability. 
  
• To collaborate on shared tasks with comments and file attachments. 
  
• To track progress through visual indicators and charts. 
  
• To manage small-to-medium projects such as: Marketing campaigns, Employee onboarding, IT system upgrades.

When to use it: 

• When managing team-based projects, collaboration and transparency are essential. 
  
• When you need a simple, visual tool for tracking deadlines and progress. 
  
• When using Outlook, Microsoft Teams, or SharePoint in together with task tracking. 
  
• When full-fledged project management software is unnecessary or overly complex. 

Benefits of Microsoft Planner: 

Benefit	Description
Smooth Integration	Straightforward integration with To Do, SharePoint, Outlook, Microsoft Teams.
Visual Task Boards	Drag-and-drop interface with a kanban board layout
Notifications & Reminders	Alerts through email and Teams for due dates and task assignments
Team Collaboration	Tasks are shareable, comment-enabled, and can be updated by multiple users
Progress Tracking	Built-in charts give a quick overview of status and bottlenecks
File Attachments	Attach images, documents, or links directly to tasks
Ease of Use	Intuitive layout with minimal learning curve
Flexible  Task Assignment	Assign tasks to one or more users, including external collaborators
Mobile & Web Access	Accessible via browsers and mobile apps, ideal for hybrid teams
Enterprise Security	Inherits Microsoft 365’s enterprise-level security protocols

Pros: 

• Integrated seamlessly with Microsoft 365 apps. 
  
• Simple, drag-and-drop interface. 
  
• Minimal training required for onboarding. 
  
• Real-time collaboration and file sharing. 
  
• Visual progress tracking (charts, boards, calendar view). 
  
• Built-in notifications and reminders. 
  
• Accessible via Microsoft Teams. 
  
• No extra cost for Microsoft 365 subscribers. 
  
• Allows multiple assignees per task. 
  
• Works on both web and mobile platforms. 

Cons: 

• Requires a Microsoft 365 business subscription (not available for personal users). 
  
• No Gantt chart or task dependencies like advanced PM(Project Management) tools. 
  
• No dedicated desktop application. 
  
• Limited customization for fields or workflows. 
  
• Not suitable for managing large-scale, complex projects. 
  
• Some terminology (like “buckets”) may be confusing for new users. 
  
• Mobile app is functional but less capable than the web version. 

Tips for Using Microsoft Planner Effectively: 

• Use Buckets Strategically – Define buckets based on task stages, priorities, or departments for clarity. 
  
• Switch Views Often – Use Board, Charts, and Schedule views to get varied insights into progress. 
  
• Add Task Details – Use checklists, due dates, attachments, and labels to keep tasks informative and actionable. 
  
• Integrate with Teams – Add Planner as a tab in Teams to enable centralized collaboration. 
  
• Monitor Charts View – Regularly check for overdue or blocked tasks to redistribute workloads. 
  
• Automate with Power Automate – Create flows to notify users or schedule events when tasks are updated. 
  
• Assign Due Dates – Ensure tasks trigger reminders and appear in schedules. 
  
• Group by Different Views – Use groupings such as “Assigned To” or “Due Date” to analyze task distribution. 
  
• Use Copy Plan – Reuse structure and task templates for recurring projects. 
  
• Archive Completed Plans – Keep your workspace clean and organized. 
  
• Use Templates – Start projects with predefined templates to save time and ensure consistency. 

Final Thoughts: 

Microsoft Planner is an intuitive and collaborative tool best suited for small to medium-sized teams already using Microsoft 365. It excels at visual task tracking, lightweight project planning, and seamless integration across the Microsoft ecosystem. While it lacks the sophistication of full project management tools, it is perfect for teams looking for simplicity, flexibility, and collaboration. 

If you have any questions you can reach out our SharePoint Consulting team here.