About our crawler

Requests from this project identify themselves as SarkariJobTrackBot/1.0, with a link to this page. It reads publicly published government recruitment notifications so candidates can search them in one place and check their own eligibility against them.

It does not spoof a browser, does not run JavaScript, does not attempt to reach anything behind a login, and does not submit forms.

• One request at a time per site — never in parallel.
• A randomised gap of 20–45 seconds between requests. The randomisation is deliberate: a request arriving exactly every N seconds is a machine signature, an irregular gap is not.
• A hard ceiling of 250 requests per site per day.
• Feeds are checked every three hours, and once overnight.
• Pages already fetched are re-checked on a lifecycle rather than a timer: a notification past its closing date is never fetched again.
• Archive pages are limited to 50 a night, so a backlog drains over weeks rather than in a burst.

Conditional requests with If-None-Match and If-Modified-Since, so an unchanged page ends at 304 before any body transfers. Responses are capped at 5 MB and abandoned after 20 seconds.

Content is hashed, so a page that has not changed is not reprocessed. In steady state most requests are 304s.

robots.txt is fetched, cached and honoured before every request, including any Crawl-delay you set. To exclude this crawler specifically:

User-agent: SarkariJobTrackBot
Disallow: /

A 429 or 503 widens the delay and it decays back slowly over hours, not immediately. Retry-After is obeyed exactly. A 403 stops requests to that site entirely rather than retrying into a block.

• Sahi Sarkari Jobs — https://www.sahisarkarijobs.in
• FreeJobAlert — https://www.freejobalert.com

Every notification page on this site links back to the page it came from.

If the crawl is causing you a problem, or you want content removed, see the removal and contact page. A request to slow down or stop is honoured without argument.