[Free Script] Google SERP Scraper + GPT Summarizer

[Mohinik]

Hello lovelies,

Been a while since I dropped something. So am here again, sharing something with you all.

You've shown a lot of love to my Keyword Clustering Script, Keyword Similarity Check and

The Problem

How can we make use of the SERPs to enhance our Content? Enter Google Scraper + Summarizer

The Magic of the script

This script will

1. Scrape Google SERPs
2. Extract top 5 results (configurable)
3. Visit those website and extract the content
4. Summarize the content using GPT 3.5 Turbo
5. Save Summaries to a JSON file with the keyword name. (So the Summar for the keyword "Best SEO Forum" will be saved to "Best SEO Forum.json"
6. Removes all done keywords from keywords.csv

How to install

1. Clone the Repository:

git clone https://github.com/iamBlogPro/Scrape-Summarize.git
cd Scrape-Summarize

Or Download the ZIP of the file

You do not have permission to view link please Log in or Register

Extract and Open it in VS Code or an IDE of your choice.

2: Install Required Packages:

pip install -r requirements.txt

3: Download Chromedriver:

Download Chromedriver from

You do not have permission to view link please Log in or Register

and place it in the same directory as the script.

Now the latest Chrome is super buggy. I recommend using Chome 110 and the chromedriver associated with it (Let me know if you want download links)

4: Edit the settings.json:

Edit a settings.json file in the root directory with the following content:

{
"openai_api_key": "your_openai_api_key",
"output_directory": "Output_Folder_Name",
"keywords_file": "keywords.csv",
"proxy_file": "proxylist.txt"
}

5: Replace "your_openai_api_key" with your actual OpenAI API key
6: In magic.py - add any domain you don't want scraped, or any extensions you don't want or any words in the URL you don't want

You can find this around line 142

def get_links_with_beautifulsoup(driver):
    logging.info("Retrieving page source for URL extraction")
    skip_domains = ["reddit.com", "istockphoto.com", "groupon.com", "youtube.com", "tiktok.com", "facebook.com", "twitter.com", "petsmart.com", "linkedin.com"]
    skip_extensions = [".gov", ".in", ".uk"]
    skip_words = ["collections", "login", "signin", "advisor", "gov"]

Using the Script

7: Prepare Input Files:

• Keywords File (keywords.csv): A CSV file containing keywords to search for. Each keyword should be on a separate line without a header.
• Proxy File (proxylist.txt): A text file containing proxies in the format iport:usernameassword, one per line.

8: Run the Script:

python magic.py

How the Script works?

1. Initialization:
   • Loads settings from settings.json.
   • Sets up logging to capture script execution details.
2. Proxy Management:
   • Loads proxies from proxylist.txt.
3. Keyword Processing:
   • Reads keywords from keywords.csv.
   • For each keyword, it performs a Google search using Selenium.
4. Content Extraction:
   • Retrieves the page source and uses BeautifulSoup to extract URLs.
   • Filters URLs based on predefined criteria (e.g., domain, extensions, keywords).
5. Data Extraction and Summarization:
   • Extracts content from each URL using Trafilatura.
   • Cleans the extracted content.
   • Summarizes the content using OpenAI's GPT-3.5 Turbo.
6. Output:
   • Saves the summarized content to JSON files in the specified output directory.
   • Updates the keywords.csv file to remove processed keywords.
7. Logging
   The script generates log files in the logs directory, which can be used to monitor its execution and troubleshoot any issues.

Features of the Script

• Proxy Management - Automatically loads and manages a list of proxies to avoid IP bans and ensure continuous scraping.
• Headless Browser - Uses Selenium with headless Chrome for efficient and seamless web scraping.
• Content Extraction - Employs Trafilatura to extract and clean textual content from web pages.
• URL Filtering - Filters out irrelevant URLs based on predefined criteria to ensure only relevant content is processed.
• Summarization - Utilizes OpenAI's GPT-3.5 Turbo to generate concise summaries of extracted content.
• Logging - Provides detailed logging for monitoring script execution and debugging purposes.
• Configuration Management - Stores configurable parameters such as API keys, file paths, and directories in a JSON settings file for easy modification.

Hope some of you can find this useful