Portal SEO Enhancements & Bot Detection Improvements — May 26, 2026

My goal for the day was to improve my portal's SEO and its ability to detect bots.  After 8 hours, I was able to do 13 commits. Out of the 13, 11 were new features or improvements and 2 were fixes. I wanted to prioritize the portal's accessibility for web crawlers and LLMs (large language models) in order to boost its ranking on the agentic web.

One of my favorite features is the new crawler tracking system, which displays which crawlers and LLMs have visited their portals. This is shown underneath the portal cover image so users can see it traffic data right away and optimize it. For the ease of managing bot detection, I expanded the bot allowlist to over 50 different bots, which includes crawlers from almost all major AI and search engine platforms.

In addition to the new crawler tracking, there are now 3D Seen-by pills, which are meant to represent the bots that interacted with the portal. This addition helps present the analytics in an easier to understand way. The pills located above the bots will say “Seen by” for everybody so there will be an improved understanding of bot engagement.

There were some challenges that came with the day. I found an issue with the bot detection logic where the Telegram bot was not being prioritized correctly due to some blockage with the Twitter bot detection. This was particularly complicated because I had to dive into user-agent strings to match the detection correctly. After several rounds of back and forth with debugging, I was able to complete it by reordering the sequence of the bot matches to ensure that the Telegram bot gets detected correctly before the Twitter bot. This solution cost me quite some time, but it was very important for accurate detection of the bots.

A better LLM-callable POST API for question asking about portals is another improvement that should be noted. It enables external LLMs to ask questions and receive answers in a structured format. Implementing this was an important step towards improving interactivity and usability of the portals. I set rate limits to mitigate possible abuses and added extensive documentation in JSON-LD format to facilitate L. Developers to better grasp the workings of the new API endpoints. This documentation was an extra effort, which futher down the line, I found to be important for functionality.

Concerning SEO, I built a dedicated sitemap that users requested. This required creating a new /portals/sitemap.xml file that will be used to index every public portal with updated timestamps. I also added nofollow attributes to event portals so that search engines won't be able to interact with those pages. These foundational SEO improvements will enable search engines and AI systems to discover our content.

I also made some user experience improvements on the home page. I adjusted the number of orbs visible in the SpaceLevelsSection and increased their opacity, creating a more colorful and visually appealing experience. Users said that the sections under the hero were flat, so the changes were intended to liven up the design of the interface.

As I look back on today, I feel a sense of achievement for the work that I have done, especially as it has all been done solo with Claude Code on VS Code. This is the first time I have experienced the full power of AI as a partner on a project. The changes push the boundaries of what the portal is capable of and bring me closer to my goal of establishing a one-man-show company valued at a billion dollars. I look forward to building more, sharing my journey, and documenting what I learn along the way, with each commit bringing me closer to that goal.