Python on Research Tool

Boolean Indexing Using Pandas

cahoover@gmail.com (Christopher Hoover) — Wed, 19 Feb 2025 00:00:00 +0000

A boolean operation returns only True or False, and can only be performed against a series. The process of boolean indexing involves two steps: first, evaluate the series to identify true/false values, then filter the dataframe based on results.

Operators

Here are the standard boolean operators:

== a : Equality comparison
~a : Negation (NOT)
a > b : Greater than
a < b : Less than
a & b : AND operator
a | b : OR operator

Filtering a Dataframe

Let’s demonstrate filtering using a dataset. First, create an evaluation series:

evaluation = df['number'] < 10

This identifies entries meeting the condition. Next, apply the filter to return matching rows:

df[evaluation]

Selecting Specific Data

You can combine conditions for more advanced filtering.

Combining OR conditions across country values:

df[(df['country'] == 'USA') | (df['country'] == 'Canada')]

Using negation with AND to exclude specific criteria while filtering by sector:

df[(~df['country'] == 'USA') & (df['sector'] == 'Technology')]

Boolean indexing enables data exploration by creating filtered datasets based on conditional logic applied to dataframe columns.

Getting Started with the Google Ads API Using Python

cahoover@gmail.com (Christopher Hoover) — Mon, 19 Feb 2024 00:00:00 +0000

Launching with the Google Ads API involves multiple steps that can feel overwhelming. This guide walks through the entire process comprehensively to help you succeed.

Here’s the complete roadmap:

Establish a Google Ads manager account (if you don’t have one)
Request a developer token through your manager account
Set up a Google Ads manager TEST account
Set up a Google Ads regular TEST account
Build sample campaigns in your regular test account
Establish a Google Cloud project
Activate the Google Ads API within your Cloud Project
Obtain OAUTH credentials from Cloud Project
Export credentials as JSON format
Download the Google Ads API library via pip
Clone the Google Ads API GitHub repository
Execute a credential script to obtain a refresh token
Transfer a YAML configuration file to your home directory
Populate the YAML with your OAUTH credentials, developer token, and refresh token
Execute your inaugural API request

Get a Google Ads Manager Account

Two distinct Google Ads account types exist:

Regular account: This functions as an operational account where you develop and execute advertising campaigns.

Manager account: This serves as a parent account for overseeing multiple regular accounts.

Manager accounts cannot run advertisements directly — they exist solely for supervision purposes. Marketing agencies, for instance, typically maintain two accounts: one regular account for their own promotions and one manager account for client management.

Without an existing manager account, establish one by visiting the Google Ads Manager Homepage.

Get a Developer Token

A developer token represents your first required credential for API access. Obtaining one is straightforward. Access your Manager account and navigate to TOOLS & SETTINGS > SETUP > API Center. (This option appears exclusively in Manager Accounts.)

Complete the application form, acknowledge the terms and conditions, and submit. Provide an active email address you monitor regularly. Once approved, your token displays as an alphanumeric string in the API Center.

Save this token in a readily accessible location, such as a text document.

Set Up a Google Ads TEST Manager Account

Initially, Google restricts API access to production accounts. For API development, you require a test environment. Create a second manager account by revisiting the Google Ads Manager Homepage and establishing another MCC using your existing email address.

This creates a test account, identifiable by the red “test account” badge displayed prominently.

Set Up a Google Ads TEST Regular Account

The documentation references generating test accounts “using the Google UI,” which can confuse newcomers. The actual process involves clicking ACCOUNTS in the left navigation menu, then selecting the “+” link.

IMPORTANT NOTE: The interface will display a complaint about account usability, present a billing warning, and appear unresponsive.

Community forums overflow with users reporting that test account creation appears unsuccessful. Google acknowledges this issue and promises resolution, though if you experience this, understand that your account exists — it’s simply concealed. Access the filter option and select “Show hidden” to reveal it.

Preserve the regular test account number in an accessible document.

Create a Test Campaign in Your Test Account

Enter your regular test account by selecting the corresponding link in your Manager account’s upper left corner.

Generate a campaign by clicking campaigns in the left sidebar and then the “+” link.

Important context about test accounts: They contain no authentic data. Although you can create campaigns, they remain unpopulated with information, including simulated data for impressions, clicks, and budgets.

Users anticipate test accounts resembling production environments with sample metrics — Facebook implements this approach, but Google does not. This disparity generates extensive community frustration documented across forums for years, yet Google has made no adjustments.

Instead, Google recommends exporting production data as CSV format and simulating API retrieval from it.

Set Up Your Google Cloud Project

This process demands additional steps. API usage requires establishing a Google Cloud project, enabling the Google Ads API within it, and configuring authorization credentials.

Begin by logging into Google Cloud using your Manager Account credentials.

Create a Google Cloud Project

In the upper left area, select the project for use with the Google Ads API. Should you lack a project, select New Project.

You may encounter billing setup prompts (particularly if you haven’t previously entered payment information). Complete this process. Unless you execute thousands of daily API calls or utilize substantial additional cloud resources, billing concerns are minimal.

Enable the Google Ads API in Your Project

Verify your project selection, then select APIs and Services from the left sidebar.

Select the + API and Services link at the page’s top.

Locate Google Ads in the search results. Upon finding it, select it and click ENABLE. When asked about application category, choose Installed App (select Web App if constructing a multi-user platform for account access — unlikely for beginners).

Create Your Credentials

Return to APIs and Services, then select credentials.

On the Credentials interface, select CREATE CREDENTIALS, then choose OAuth client ID.

Select “Other” as the application type.

Two credentials display: a client ID and corresponding secret. Store both values securely in a text document.

After closing this dialog, only the Client ID appears on the main credentials page. To access your secret subsequently, select your OAUTH credentials by name.

Download Your Credentials as a JSON File

On the credentials page, a download icon appears to the right of your OAUTH credentials. Selecting this downloads credentials in JSON format.

The downloaded file carries an extensive filename. For practical purposes, rename it to something manageable like “secrets.json”. Move this file from your downloads folder to an accessible location, such as your home directory.

Get the API Library on Your Computer

Now comes the exciting part — retrieving the Google Ads API locally and executing your inaugural call.

Start by entering a Python virtual environment.

Within your virtual environment, install setuptools, a prerequisite for the API library.

Install the library using pip by typing this command:

pip install google-ads

The library includes extensive example code deserving easy access. Cloning the API library Git repository directly into your home directory proves helpful:

git clone git@github.com:googleads/google-ads-python.git

Navigate into the library and execute the setup test to verify proper installation:

python setup.py test

There’s One Last Credential We Need to Get

Yes, there’s another credential required. Consider this comprehensive authentication process as a competitive barrier. The final credential — a refresh token — allows refreshing your credentials upon expiration.

Navigate to the google-ads-python directory created during repository cloning, then proceed to the examples/authentication folder:

cd examples/authentication

Three files reside in this directory. The target file is authenticate_in_standalone_application.py. Execute it, specifying your previously downloaded JSON file location. Type this command (substituting /path/to/secrets.json with your actual file path and name):

./authenticate_in_standalone_application.py --client_secrets_path=/path/to/secrets.json

This script generates a URL.

Launch a private or incognito browser session. This URL carries sensitive information — you shouldn’t preserve it in your browser history. Transfer the URL into your private browser session.

Select allow.

An authorization code becomes visible.

Transfer the verification code to your command line session running authenticate_in_standalone_application.py and press enter:

After approving the token enter the authorization code here: ****

Upon successful entry:

After approving the token enter the authorization code here: ****
Your refresh token is: ****

Document this refresh token in an accessible location.

Rounding Up All Your Credentials and Getting Them Configured

You should now possess a collection of credentials stored somewhere accessible. Time to implement them.

By default, the Google Ads API searches for credentials in a YAML file within the home directory labeled google-ads.yaml. This file resides within your cloned google-ads-python directory from GitHub. Your tasks involve:

Duplicate the YAML file
Relocate it to your home directory
Populate it with all credentials you’ve accumulated

After transferring the file to your home directory, modify it with your appropriate information and save it.

Make Your First API Call!

Finally, the moment arrives. Execute an API call retrieving the name of your test account campaign.

First, navigate to the relevant example script:

cd google-ads-python/examples/basic_operations

Execute get_campaigns.py, providing your test account number as an argument:

python get_campaigns.py -c1234567890

Observe how the account ID transfers without spacing or dashes. Assuming correct configuration, you’ll receive output resembling:

Campaign with ID xxxxxxxxxxx and name "Campaign-Name-1" was found.

Scraping TripAdvisor reviews using Python

cahoover@gmail.com (Christopher Hoover) — Mon, 19 Feb 2024 00:00:00 +0000

We’re going to use scrape review data from Tripadvisor, which would be helpful if you wanted to do a NLP analysis of reviews as a way to keep your finger on the pulse of customer satisfaction.

Selenium is a tool that lets you control your desktop browser programmatically. Tripadvisor requires a “read more” button to be clicked to reveal the entire review, and Selenium is a good tool to easily get this done.

Install Selenium

We’ll assume you’re using Anaconda for environment management. First, activate an environment with

conda activate [environment]

Install Selenium with:

conda install -c conda-forge selenium

Import packages

The first thing we’ll do is import the required packages. There aren’t many:

import csv #This package lets us save data to a csv file

from selenium import webdriver #The Selenium package we'll need

import time #This package lets us pause execution for a bit

Define variables

We’ll need to set up some variables.

First, define a path and file to store data. Here I’m just putting it on my Mac’s desktop

path_to_file = "/Users/chris/Desktop/HotelReviews.csv"

Next, define the number of review pages to scrape. There might be hundreds of pages of reviews, depending on the property. For this demonstration we’ll keep it easy and just grab three pages.

pages_to_scrape = 3

Next, define the Tripadvisor URL. We’ll use the Hilton Waikiki resort page, because it’s a super nice hotel!

url = "https://www.tripadvisor.com/Hotel_Review-g60982-d209422-Reviews-Hilton_Waikiki_Beach-Honolulu_Oahu_Hawaii.html"

Get Selenium on the right page and set up the CSV

Let’s launch Selenium and have it point the browser to the URL we’re using. This example uses the Safari browser, but you can remote control Chrome (or any other browser) if you prefer.

# import the webdriver
driver = webdriver.Safari()
driver.get(url)

We’ll also get a CSV set up to contain the data we retrieve

# open the file to save the review
csvFile = open(path_to_file, 'a', encoding="utf-8")
csvWriter = csv.writer(csvFile)

Programmatically grab the reviews, one after the other

Now it’s time to tell Selenium to grab the reviews. Because Tripadvisor reviews are truncated, we’ll be sure and click the “read more” button for all reviews first.

# change the value inside the range to save the number of reviews we're going to grab
for i in range(0, pages_to_scrape):

    # give the DOM time to load
    time.sleep(2)

    # Click the "expand review" link to reveal the entire review.
    driver.find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

    # Now we'll ask Selenium to look for elements in the page and save them to a variable. First lets define a  container that will hold all the reviews on the page. In a moment we'll parse these and save them:
    container = driver.find_elements_by_xpath("//div[@data-reviewid]")

    # Next we'll grab the date of the review:
    dates = driver.find_elements_by_xpath(".//div[@class='_2fxQ4TOx']")

   # Now we'll look at the reviews in the container and parse them out

    for j in range(len(container)): # A loop defined by the number of reviews

        # Grab the rating
        rating = container[j].find_element_by_xpath(".//span[contains(@class, 'ui_bubble_rating bubble_')]").get_attribute("class").split("_")[3]
        # Grab the title
        title = container[j].find_element_by_xpath(".//div[contains(@data-test-target, 'review-title')]").text
        #Grab the review
        review = container[j].find_element_by_xpath(".//q[@class='IRsGHoPm']").text.replace("\n", "  ")
        #Grab the data
        date = " ".join(dates[j].text.split(" ")[-2:])

        #Save that data in the csv and then continue to process the next review
        csvWriter.writerow([date, rating, title, review])

    # When all the reviews in the container have been processed, change the page and repeat
    driver.find_element_by_xpath('.//a[@class="ui_button nav next primary "]').click()

# When all pages have been processed, quit the driver
driver.quit()

Conclusion

That’s it! A very simple program. You’ll now have a csv that contains all the reviews you retrieved.