| Title: | Get the Category of Content Hosted by a Domain |
|---|---|
| Description: | Get the category of content hosted by a domain. Use Shallalist (service discontinued), 'VirusTotal' (which provides access to lots of services) <https://www.virustotal.com/>, 'DMOZ' <https://archive.org/details/dmoz-rdf-20150327>, University Domain list <https://github.com/Hipo/university-domains-list>, 'OpenAI' 'GPT' models, 'Anthropic' 'Claude' models, or validated machine learning classifiers based on 'Shallalist' data to learn about the kind of content hosted by a domain. |
| Authors: | Gaurav Sood [aut, cre] |
| Maintainer: | Gaurav Sood <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.5.0 |
| Built: | 2026-10-09 12:04:17 UTC |
| Source: | https://github.com/cran/rdomains |
Want to know what kind of content is carried on a domain? Get the results quickly using rdomains. The package provides access to virustotal API, shalla, aws, OpenAI GPT models, Anthropic Claude models, and validated ML model based off shallalist data to predict content of a domain.
To learn how to use rdomains, see this vignette: ../doc/rdomains.html.
Gaurav Sood
Uses a validated ML model that uses keywords in the domain name and suffix to predict probability that the domain hosts adult content. For more information see https://github.com/themains/keyword_porn
adult_ml1_cat(domains = NULL)adult_ml1_cat(domains = NULL)
domains |
required; string; vector of domain names |
data.frame with original list and content category of the domains
## Not run: adult_ml1_cat("http://www.google.com") ## End(Not run)## Not run: adult_ml1_cat("http://www.google.com") ## End(Not run)
Fetches category of content hosted by a domain using Anthropic's Claude API. The function uses Claude models to classify domains into specified categories.
claude_cat( domains = NULL, api_key = NULL, categories = NULL, model = "claude-3-haiku-20240307", rate_limit = 0.5 )claude_cat( domains = NULL, api_key = NULL, categories = NULL, model = "claude-3-haiku-20240307", rate_limit = 0.5 )
domains |
vector of domain names |
api_key |
Anthropic API key. If not provided, looks for ANTHROPIC_API_KEY or CLAUDE_API_KEY environment variable |
categories |
vector of categories to classify into. If NULL, uses default web categories |
model |
Claude model to use (default: "claude-3-haiku-20240307" for cost efficiency) |
rate_limit |
delay in seconds between API calls (default: 0.5) |
data.frame with original list and content category of the domain
## Not run: claude_cat("google.com") claude_cat(c("google.com", "facebook.com")) claude_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other")) ## End(Not run)## Not run: claude_cat("google.com") claude_cat(c("google.com", "facebook.com")) claude_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other")) ## End(Not run)
Every requested domain comes back, in order, whether or not it was reachable. Failures carry a code from [fetch_error_codes()] rather than a bare 'NA'.
collect_content( domains = NULL, delay = 1, timeout = 10, max_bytes = 2 * 1024^2, obey_robots = TRUE, max_crawl_delay = 30, max_redirects = 5, user_agent = rdomains_user_agent() )collect_content( domains = NULL, delay = 1, timeout = 10, max_bytes = 2 * 1024^2, obey_robots = TRUE, max_crawl_delay = 30, max_redirects = 5, user_agent = rdomains_user_agent() )
domains |
Character vector of domains or URLs. |
delay |
Minimum seconds between requests to the same host. 'Crawl-delay' overrides this upward. |
timeout |
Per-request timeout, seconds. |
max_bytes |
Cap on the response body actually read. |
obey_robots |
Whether to fetch and honour robots.txt. Turning this off is discouraged and is your responsibility, not the package's. |
max_crawl_delay |
Skip a host that asks for a longer delay than this rather than sleeping on it. |
max_redirects |
Maximum redirect hops to follow. Every hop is re-validated, so a redirect cannot be used to reach an address the first check refused. |
user_agent |
Override the identifying user-agent. |
The crawler identifies itself as 'rdomains/<version>', obeys 'robots.txt' including 'Crawl-delay', spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.
A tibble with one row per input: 'domain_name', 'status', 'stage', 'error_code', 'retryable', 'http_status', 'final_url', 'fetched_at', 'content_bytes', 'title', 'description', 'lang', 'text', 'n_tokens', 'page_state', 'block_vendor', 'robots_allowed', 'source_last_published'.
[fetch_report()] to summarise the run, [page_signals()] for what the page states are, [source_vintage()] for how a live fetch compares with the static lists.
## Not run: res <- collect_content(c("example.com", "wikipedia.org")) fetch_report(res) # retry only what is worth retrying again <- collect_content(res$domain_name[res$retryable]) ## End(Not run)## Not run: res <- collect_content(c("example.com", "wikipedia.org")) fetch_report(res) # retry only what is worth retrying again <- collect_content(res$domain_name[res$retryable]) ## End(Not run)
Fetches category (or categories) of content hosted by a domain according to DMOZ.
The function checks if path to the DMOZ file is provided by the user.
If not, it looks for dmoz_domain_cateory.csv in the working directory. It also returns
results for prominent subdomains.
dmoz_cat(domains = NULL, use_file = NULL)dmoz_cat(domains = NULL, use_file = NULL)
domains |
vector of domain names |
use_file |
path to the dmoz file, which can be downloaded using |
DMOZ closed in March 2017 and the snapshot shipped here is from 2015, so these labels
describe the domain as it was a decade ago. The returned source_last_published
column carries that vintage, and a warning is issued once per session. See
source_vintage.
data.frame with the original list, the content category of the domain, and the vintage of the list that supplied it
source_vintage for the provenance of every category source
## Not run: dmoz_cat(domains = "http://www.google.com") dmoz_cat(domains = c("http://www.google.com", "http://plus.google.com")) ## End(Not run)## Not run: dmoz_cat(domains = "http://www.google.com") dmoz_cat(domains = c("http://www.google.com", "http://plus.google.com")) ## End(Not run)
The closed set of 'error_code' values that [collect_content()] and [pie_cat()] can return, with the pipeline stage each belongs to and whether it is worth retrying.
fetch_error_codes()fetch_error_codes()
These strings are shared verbatim with the Python package 'piedomains', so results from the two can be aggregated together.
A tibble with columns 'code', 'stage', 'retryable' and 'description'.
[fetch_report()] to summarise an actual run, [source_vintage()] for the provenance of the static label sources.
fetch_error_codes() # which failures are worth trying again? subset(fetch_error_codes(), retryable)fetch_error_codes() # which failures are worth trying again? subset(fetch_error_codes(), retryable)
A pure function over the returned tibble, deliberately **not** a stored summary. A snapshot goes stale the moment you filter the rows; recomputing means 'fetch_report(subset(x, retryable))' is always correct.
fetch_report(x)fetch_report(x)
x |
A tibble from [collect_content()] or [pie_cat()]. |
A one-row tibble: 'n', 'ok', 'failed', 'retryable', and list-columns 'by_stage', 'by_reason' and 'missing' (the domains that produced nothing).
rows <- data.frame( domain_name = c("a.com", "b.com", "c.com"), status = c("ok", "failed", "failed"), stage = c("infer", "fetch", "process"), error_code = c(NA, "timeout", "thin_content"), retryable = c(FALSE, TRUE, FALSE) ) fetch_report(rows)rows <- data.frame( domain_name = c("a.com", "b.com", "c.com"), status = c("ok", "failed", "failed"), stage = c("infer", "fetch", "process"), error_code = c(NA, "timeout", "thin_content"), retryable = c(FALSE, TRUE, FALSE) ) fetch_report(rows)
Downloads archived DMOZ (Open Directory Project) data. DMOZ was discontinued in March 2017. This function downloads our preserved copy of the final DMOZ dataset. For more details, check: https://github.com/themains/rdomains/tree/master/data-raw/dmoz/
get_dmoz_data(outdir = ".", overwrite = FALSE)get_dmoz_data(outdir = ".", overwrite = FALSE)
outdir |
Optional; folder to which you want to save the file; Default is same folder |
overwrite |
Optional; default is FALSE. If TRUE, the file is overwritten. |
https://archive.org/details/dmoz-rdf-20150327
## Not run: get_dmoz_data() ## End(Not run)## Not run: get_dmoz_data() ## End(Not run)
Shallalist service was discontinued in January 2022. This function downloads
the last archived copy (from 1/14/22) that we have preserved on GitHub.
The original service at shallalist.de is no longer available.
Downloads, unzips and saves the final version of shallalist data. By default, saves shalla data
as shalla_domain_category.csv.
get_shalla_data(outdir = "./", overwrite = FALSE)get_shalla_data(outdir = "./", overwrite = FALSE)
outdir |
Optional; folder to which you want to save the file; Default is same folder |
overwrite |
Optional; default is FALSE. If TRUE, the file is overwritten. |
https://web.archive.org/web/20210502020725/http://www.shallalist.de/
## Not run: get_shalla_data() ## End(Not run)## Not run: get_shalla_data() ## End(Not run)
Downloads the latest version of Steven Black's unified hosts file. The hosts file contains domains known for serving ads, malware, and tracking.
get_stevenblack_data(outdir = "./", variant = "base", overwrite = FALSE)get_stevenblack_data(outdir = "./", variant = "base", overwrite = FALSE)
outdir |
Optional; folder to which you want to save the file; Default is current directory |
variant |
Optional; which variant to download. Options: "base", "porn", "social", "gambling", "all" |
overwrite |
Optional; default is FALSE. If TRUE, the file is overwritten. |
https://github.com/StevenBlack/hosts
## Not run: get_stevenblack_data() get_stevenblack_data(variant = "all") ## End(Not run)## Not run: get_stevenblack_data() get_stevenblack_data(variant = "all") ## End(Not run)
ML Model
glm_shallaglm_shalla
A list
Gaurav Sood
ML model based on shallalist using keywords and domain suffixes,
Extract text, title, description and language from HTML
html_text_content(html)html_text_content(html)
html |
Raw HTML as a single string. |
A list with 'text' (cleaned, lowercased), 'title', 'description' and 'lang'. Missing elements are 'NA_character_'; 'text' is '""' when nothing could be extracted.
[page_signals()], which uses this to decide whether a page is classifiable.
html <- "<html lang='en'><head><title>Example</title> <meta name='description' content='A demo page'> <script>ignored()</script></head> <body><p>Hello World</p><style>p{}</style></body></html>" html_text_content(html)html <- "<html lang='en'><head><title>Example</title> <meta name='description' content='A demo page'> <script>ignored()</script></head> <body><p>Hello World</p><style>p{}</style></body></html>" html_text_content(html)
Based on a slightly amended version of the regular expression used to classify news, and non-news in: “Exposure to ideologically diverse news and opinion on Facebook” by Bakshy, Messing, and Adamic. Science. 2015.
not_news(url_list = NULL)not_news(url_list = NULL)
url_list |
vector of URLs |
Amendment: sport rather than sports
URL containing any of the following words is classified as soft news: "sport|entertainment|arts|fashion|style|lifestyle|leisure|celeb|movie|music|gossip|food|travel|horoscope|weather|gadget"
URL containing any of following words is classified as hard news: "politi|usnews|world|national|state|elect|vote|govern|campaign|war|polic|econ|unemploy|racis|energy|abortion|educa|healthcare|immigration"
Note that it is based on patterns existing in a small set of domains. See paper for details.
data.frame with 3 columns: url, not_news, news
https://www.science.org/doi/10.1126/science.aaa1160
## Not run: not_news("http://www.bbc.com/sport") not_news(c("http://www.bbc.com/sport", "http://www.washingtontimes.com/news/politics/")) ## End(Not run)## Not run: not_news("http://www.bbc.com/sport") not_news(c("http://www.bbc.com/sport", "http://www.washingtontimes.com/news/politics/")) ## End(Not run)
Fetches category of content hosted by a domain using OpenAI's chat completion API. The function uses GPT models to classify domains into specified categories.
openai_cat( domains = NULL, api_key = NULL, categories = NULL, model = "gpt-4o-mini", rate_limit = 0.5 )openai_cat( domains = NULL, api_key = NULL, categories = NULL, model = "gpt-4o-mini", rate_limit = 0.5 )
domains |
vector of domain names |
api_key |
OpenAI API key. If not provided, looks for OPENAI_API_KEY environment variable |
categories |
vector of categories to classify into. If NULL, uses default web categories |
model |
OpenAI model to use (default: "gpt-4o-mini" for cost efficiency) |
rate_limit |
delay in seconds between API calls (default: 0.5) |
data.frame with original list and content category of the domain
## Not run: openai_cat("google.com") openai_cat(c("google.com", "facebook.com")) openai_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other")) ## End(Not run)## Not run: openai_cat("google.com") openai_cat(c("google.com", "facebook.com")) openai_cat("google.com", categories = c("search", "social", "ecommerce", "news", "other")) ## End(Not run)
Inspects already-fetched HTML for the three things that are not a classifiable site: an anti-bot interstitial, a domain-parking placeholder, and a server's "nothing here" page. No network access.
page_signals(html, text = NULL, domain = "", status = NULL)page_signals(html, text = NULL, domain = "", status = NULL)
html |
Raw response body. |
text |
Extracted page text. If 'NULL', derived from 'html'. |
domain |
The domain requested, used to spot a page whose only content is its own name. |
status |
HTTP status code, if known. |
'parked' and 'unavailable' are **answers, not failures** – they are facts about the domain, plainly stated in the page, that a caller can act on. They are also free: no classification service needs to be consulted.
A one-row tibble: 'page_state' (one of '"content"', '"blocked"', '"parked"', '"unavailable"', '"thin"'), 'blocked', 'block_vendor', 'block_reason', 'parked', 'unavailable', 'thin', 'n_tokens'.
[fetch_error_codes()] for how these map onto run outcomes.
# A Cloudflare challenge, not a website page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>") # A parking page page_signals("<html><body>This domain is for sale. Inquire now.</body></html>") # A real page that merely embeds reCAPTCHA is not blocked page_signals(paste0("<html><title>Reddit</title><body>", paste(rep("real discussion content", 200), collapse = " "), "<script src='recaptcha/api.js'></script></body></html>"))# A Cloudflare challenge, not a website page_signals("<html><title>Just a moment...</title><body>cf_chl_opt</body></html>") # A parking page page_signals("<html><body>This domain is for sale. Inquire now.</body></html>") # A real page that merely embeds reCAPTCHA is not blocked page_signals(paste0("<html><title>Reddit</title><body>", paste(rep("real discussion content", 200), collapse = " "), "<script src='recaptcha/api.js'></script></body></html>"))
Fetches category of content hosted by a domain according to Shalla.
The function checks if path to the shalla file is provided by the user.
If not, it looks for shalla_domain_category.csv in the working directory.
shalla_cat(domains = NULL, use_file = NULL)shalla_cat(domains = NULL, use_file = NULL)
domains |
vector of domain names |
use_file |
path to the latest shallalist file downloaded using |
Shallalist stopped publishing in 2022, so these labels describe the domain as it was
then or earlier, not as it is now. The returned source_last_published column
carries that vintage, and a warning is issued once per session. See
source_vintage.
data.frame with the original list, the content category of the domain, and the vintage of the list that supplied it
source_vintage for the provenance of every category source
## Not run: shalla_cat(domains = "http://www.google.com") ## End(Not run)## Not run: shalla_cat(domains = "http://www.google.com") ## End(Not run)
Every lookup in this package answers from a static list, and those lists have very different vintages. Two of them are no longer published at all. Use this to see which answers can be trusted to describe a domain as it is today.
source_vintage(source = NULL)source_vintage(source = NULL)
source |
Optional; one of |
A tibble with one row per source: key, source,
last_published, status, successor and note.
source_vintage() source_vintage("dmoz")source_vintage() source_vintage("dmoz")
Classifies domains based on Steven Black's unified host list which blocks ads, malware, and tracking domains. The function checks if a domain appears in the blocklist and categorizes it accordingly.
stevenblack_cat(domain = NULL, use_file = NULL)stevenblack_cat(domain = NULL, use_file = NULL)
domain |
domain names as character vector |
use_file |
path to a local Steven Black hosts file. If NULL, downloads from GitHub once per session and reuses it |
Steven Black's host list is a consolidated list from multiple sources including adaway.org, mvps.org, malwaredomainlist.com, and someonewhocares.org.
Unlike the Shallalist and DMOZ lookups, this list is actively maintained, so the
returned source_last_published is the fetched file's own modification date
rather than a constant. See source_vintage.
data.frame with the original domain name, the category, and the date of the list that supplied it
https://github.com/StevenBlack/hosts
source_vintage for the provenance of every category source
## Not run: stevenblack_cat("doubleclick.net") stevenblack_cat(c("google.com", "googleadservices.com", "malware-example.com")) ## End(Not run)## Not run: stevenblack_cat("doubleclick.net") stevenblack_cat(c("google.com", "googleadservices.com", "malware-example.com")) ## End(Not run)
Fetches university domain json from: https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json
uni_cat(domains = NULL)uni_cat(domains = NULL)
domains |
vector of domain names |
data.frame with original list and all the other columns from the university json
## Not run: uni_cat(domains = "http://www.google.com") ## End(Not run)## Not run: uni_cat(domains = "http://www.google.com") ## End(Not run)
Returns category of content from multiple security vendors using the VirusTotal API v3. The function retrieves domain analysis results including categories from various security services. Not all services will have categories for all domains.
virustotal_cat(domains = NULL, apikey = NULL)virustotal_cat(domains = NULL, apikey = NULL)
domains |
domain names as character vector |
apikey |
virustotal API key |
Get the API Access Key from https://www.virustotal.com/. Either pass the API Key to the function
or set the environmental variable: VirustotalToken. Environment variables persist within
a R session.
data.frame with domain and VirusTotal analysis results
https://docs.virustotal.com/reference/domains
## Not run: virustotal_cat("http://www.google.com") virustotal_cat(c("google.com", "facebook.com")) ## End(Not run)## Not run: virustotal_cat("http://www.google.com") virustotal_cat(c("google.com", "facebook.com")) ## End(Not run)