The Bynamely crawler
If you found this URL in your access logs, this page is for you. It describes exactly what we fetch, how often, and how to stop us.
How to identify us
Every request we make carries this User-Agent, unchanged:
bynamely/0.1 (+https://bynamely.com/about/crawler)We never send a browser User-Agent. If a request claiming to be a browser is reaching you, it is not us — and a publisher who blocks our real User-Agent is telling us not to fetch, which we treat as the answer.
What we fetch
RSS and Atom feeds only. We request the feed URL you publish, and nothing else. We do not request article pages, we do not follow links out of a feed, we do not render JavaScript, and we do not extract anything from your site beyond what the feed itself contains.
We store the headline, link, timestamp and whatever summary the feed carries. When we show an update built from your reporting, we name your publication and link to your page — the link is the point, not an afterthought.
How often
Feeds sit in one of three tiers. The interval is measured from our last request, and a feed that fails is backed off further.
- High
- every 15 minutes
- Normal
- every hour
- Low
- every 6 hours
We send If-None-Match and If-Modified-Since on every poll, so an unchanged feed costs you a 304 with no body. We cap response bodies at 8 MB and time out after 10 seconds.
How to block us
Return a 403 to our User-Agent, or write to us. We read a refusal as an answer, not an obstacle: a feed that blocks us is removed from our source list rather than retried with a different User-Agent or fetched some other way.
We do not currently parse robots.txt — we fetch only feed URLs that were added to our source list by hand, never by discovery, so there is no crawl to constrain. We would rather say that plainly than claim a compliance we have not built. If you have disallowed us there and want to be sure, email us and we will remove the feed.
Getting removed, or getting in touch
Write to crawler@bynamely.com and tell us the feed. We will remove it, and you do not have to give a reason.
The same address reaches us for a correction. If we have published something built on your reporting and got it wrong, we will correct or withdraw it at its original URL rather than deleting the page — more on how that works.