How it works
The Web Connector scrapes sites based on a base URL.- It only indexes files from the same domain and containing the same base path.
- It will index pages reachable via hyperlinks from the base URL.
- The text contents are cleaned up via some heuristics and some metadata such as the page Title is extracted.
Setting up
Authorization
- As long as the page is reachable, no additional authorization is necessary.
Indexing
1
Open Web connector
Navigate to the Admin Panel and select the Web Connector.
2
Enter base URL and index
Input the base URL to index and click on Index.