Testing Python Lambda Functions Locally

This article introduces how to run Python Lambda functions locally for debugging purposes, using the python–lambda–local module.

This is a continuation of the following article, so I recommend first trying to run your function on AWS Lambda as described there:

https://gonkunblog.com/lambda-selenium-setup/1761/

 

Prerequisites

My local environment is as follows:

• pyenv: stable 2.3.31
• python: 3.7.3
• pip 23.2.1
• selenium 4.1.0

 

If you have trouble setting up Python 3.7.3, please refer to this article:

https://gonkunblog.com/python-3-7-3-install-error/1755/

 

 

Final Directory Structure

The final directory structure will look like this:

~/D/g/selenium_test ❯❯❯ tree .                                                                                                                                  main ✭ ✖ ✱
.
├── app
│   └── lambda_function.py
├── chromedriver
├── headless-chromium
└── test
    └── event.json

 

Here’s the source code we’ll use (lambda_function.py):

#coding: UTF-8
import os
import time
import pytz
from selenium import webdriver
from selenium.webdriver.chrome.options import Options  # Required to use options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')

CHROMEDRIVER_PATH = os.environ["CHROMEDRIVER_PATH"]
HEADLESS_CHROMIUM_PATH = os.environ["HEADLESS_CHROMIUM_PATH"]

def lambda_handler(event, context):
  options = webdriver.ChromeOptions()
  options.add_argument("--disable-gpu")
  options.add_argument("--hide-scrollbars")
  options.add_argument("--ignore-certificate-errors")
  options.add_argument("--window-size=880x996")
  options.add_argument("--no-sandbox")
  options.add_argument("--homedir=/tmp")

  if "env" in event and event["env"] == "local":
    print("This is local env.")
  else:
    options.add_experimental_option("w3c", True)
    options.add_argument("--headless")
    options.add_argument("--single-process")  # if enable this option, it doesn't work in local
    options.binary_location = HEADLESS_CHROMIUM_PATH

  driver = webdriver.Chrome(
    executable_path=CHROMEDRIVER_PATH,
    options=options
  )

  driver.get("https://www.google.com/")
  driver.quit()

 

The event.json file is as follows, indicating that the execution environment is local:

{
  "env": "local"
}

 

When you run the following command, the Lambda function will start in the local environment:

python-lambda-local -t 100 -f lambda_handler app/lambda_function.py test/event.json

 

Steps to Run Lambda Locally

Download the chromedriver for your local environment.
(Skip this if you’ve already done it)

We won’t use headless-chromium this time, but let’s download it just in case.

curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip

curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip

unzip -o chromedriver.zip -d .

unzip -o headless-chromium.zip -d .

Install python-lambda-local:

~/D/g/selenium_test ❯❯❯ pip install python-lambda-local

 

Create the ./test/event.json file. This will be passed as the event argument to the lambda_handler function:

{
  "env": "local"
}

 

Next, prepare lambda_handler.py:

#coding: UTF-8
import os
import time
import pytz
from selenium import webdriver
from selenium.webdriver.chrome.options import Options  # Required to use options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from datetime import datetime, timedelta
import warnings
warnings.filterwarnings('ignore')

CHROMEDRIVER_PATH = os.environ["CHROMEDRIVER_PATH"]
HEADLESS_CHROMIUM_PATH = os.environ["HEADLESS_CHROMIUM_PATH"]

def lambda_handler(event, context):
  options = webdriver.ChromeOptions()
  options.add_argument("--disable-gpu")
  options.add_argument("--hide-scrollbars")
  options.add_argument("--ignore-certificate-errors")
  options.add_argument("--window-size=880x996")
  options.add_argument("--no-sandbox")
  options.add_argument("--homedir=/tmp")

  if "env" in event and event["env"] == "local":
    print("This is local env.")
  else:
    options.add_experimental_option("w3c", True)
    options.add_argument("--headless")
    options.add_argument("--single-process")  # if enable this option, it doesn't work in local
    options.binary_location = HEADLESS_CHROMIUM_PATH

  driver = webdriver.Chrome(
    executable_path=CHROMEDRIVER_PATH,
    options=options
  )

  driver.get("https://www.google.com/")
  driver.quit()

 

In the if statement, we’re skipping some options that aren’t needed for local environments.
(Including these options would prevent Selenium from starting locally)

Now let’s set the environment variables used in the program.

Specify the two files we downloaded earlier:

export HEADLESS_CHROMIUM_PATH=[path to file]/headless-chromium

export CHROMEDRIVER_PATH=[path to file]/chromedriver

 

Run the Lambda function from the local environment.

We’re passing the timeout duration with the -t option:

~/D/g/selenium_test ❯❯❯ python-lambda-local -t 100 -f lambda_handler app/lambda_function.py test/event.json                                                         main ✭ ✖ ✱
[root - INFO - 2023-10-25 00:09:35,835] Event: {'env': 'local'}
[root - INFO - 2023-10-25 00:09:35,836] START RequestId: 9e7527bd-09e9-4c19-af65-21611df6b9b9 Version: 
This is local env.
[root - INFO - 2023-10-25 00:09:41,684] END RequestId: 9e7527bd-09e9-4c19-af65-21611df6b9b9
[root - INFO - 2023-10-25 00:09:41,687] REPORT RequestId: 9e7527bd-09e9-4c19-af65-21611df6b9b9  Duration: 5077.01 ms
[root - INFO - 2023-10-25 00:09:41,687] RESULT:
None

 

We’ve confirmed that Lambda works locally.

This allows you to test your function without having to run it on AWS Lambda every time.

The next step to improve usability would be to manage this source code in a GitHub repository and set up automatic deployment to Lambda when changes are made to the master branch.
(I’ll cover that in a separate article)

 

Thank you for reading this far.