Web Scraping with RPA

Web scraping is the process of extracting data from websites automatically. Instead of a human reading a web page and copying values into a spreadsheet, the bot navigates to the page, identifies the data elements, reads their values, and stores the extracted data. RPA makes web scraping accessible without requiring advanced programming knowledge.

Common Web Scraping Use Cases

• Extracting product prices from supplier websites for price comparison
• Collecting shipping status updates from courier portals
• Pulling competitor pricing from e-commerce sites
• Extracting job postings from HR portals for analysis
• Gathering publicly available financial data from government or stock market sites
• Reading data from web-based ERP or CRM portals that have no data export feature

Types of Web Data

Static Web Pages

The page content is already present in the HTML when the page loads. Simple selector-based extraction works reliably here. Example: a product detail page on a supplier website.

Dynamic Web Pages

Content loads after the page through JavaScript or AJAX calls. The bot must wait for the content to fully appear before extracting it. Examples: search results pages that load as you scroll, dashboards that fetch live data.

Web Scraping Methods in UiPath

Method 1: Get Text (Single Value)

Use Get Text to extract one specific value from a web page — such as a product price, a stock quantity, or a status label.

 EXAMPLE: Extract current USD/INR exchange rate from a finance site

 Navigate to: https://financialdatasite.com/currency/usd-inr
 Get Text [Rate Label] → exchangeRate (String variable)
 Result: exchangeRate = "83.47"

Method 2: Data Scraping (Tables and Repeated Elements)

Data Scraping is UiPath's built-in wizard for extracting structured data — like a full table or a list of repeated elements. You indicate the first element, indicate the second element to define the pattern, and UiPath automatically identifies all matching elements on the page (and across multiple pages if pagination is present).

 EXAMPLE: Scrape all products from a supplier catalogue

 1. Click "Data Scraping" in UiPath toolbar
 2. Click on the first Product Name in the list
 3. Click on the second Product Name to define the pattern
 4. Wizard detects all matching elements on page
 5. Add columns: Price, SKU, Stock Status
 6. Set "Move to next page" selector for pagination
 7. Extract → stored in DataTable: dt_products

 Result: DataTable with 500 rows of product data,
         scraped across 10 pages automatically.

Method 3: Web Browser Activities (Manual Scraping)

For pages with non-standard structures, you use individual activities: Navigate To, Get Text, Get Attribute, Find Children — combining them manually to extract the data you need.

Handling Pagination

Many web pages split data across multiple pages. The bot must click "Next Page" and repeat the extraction until all pages are processed.

 PAGINATION LOOP PATTERN:
 ─────────────────────────────────────────────────
 pageNumber = 1
 allData = new DataTable()

 WHILE [Next Page button exists]
   ├── Scrape current page → currentPageData
   ├── Merge currentPageData into allData
   ├── Click [Next Page Button]
   ├── Wait for page to load (Element Exists check)
   └── pageNumber += 1

 Write allData to Excel
 ─────────────────────────────────────────────────

Waiting for Dynamic Content

Dynamic web pages load content asynchronously. If the bot tries to read data before it appears, it gets an empty result or an error. Use these strategies:

• Element Exists with Timeout: Wait up to 30 seconds for a specific element (like a table row or a status label) to appear before reading.
• Wait for Page Load: UiPath's built-in wait that pauses until the browser reports the page is fully loaded.
• Polling Loop: Check every few seconds whether the expected element has appeared, up to a maximum number of attempts.

Web Scraping Diagram: Product Price Monitor

 GOAL: Monitor prices of 50 products on a supplier website daily

 TRIGGER: Scheduled daily at 7:00 AM

 BOT FLOW:
 ──────────────────────────────────────────────────
 Read product list from Excel (50 product URLs)
         │
         ▼
 FOR EACH product URL in list:
   ├── Navigate to product URL
   ├── Wait for price element to appear
   ├── Get Text [Price Label] → currentPrice
   ├── Get Text [Stock Status] → stockStatus
   ├── Read yesterday's price from Excel → previousPrice
   ├── Calculate: priceChange = currentPrice - previousPrice
   ├── Write to Excel: Date | Product | Price | Change | Stock
   └── IF priceChange > 10%: Add to "Alerts" list
         │
         ▼
 IF Alerts list is not empty:
   Send email to procurement team with price change report
         │
         ▼
 Save Excel → End

Legal and Ethical Considerations

Web scraping must be done responsibly:

• Check the website's Terms of Service before scraping — some sites prohibit automated data collection.
• Do not scrape personal data about individuals without consent — this may violate privacy laws.
• Rate-limit your bot's requests to avoid overloading the target server. Add delays between page loads.
• Use official APIs when available — they are more reliable and legally safer than UI scraping.
• Scraping publicly available data for internal business analysis is generally acceptable. Republishing or selling scraped data is risky and may not be legal.

Common Web Scraping Challenges

Summary

Web scraping with RPA extracts data from web pages automatically — from simple single-value reads using Get Text, to structured table extraction using the Data Scraping wizard, to multi-page pagination loops. Key challenges include dynamic content, CAPTCHAs, and changing page layouts. Always respect the website's terms of service and rate-limit your bot to avoid server overload. When an official API is available, prefer it over UI scraping for reliability and compliance.