Automating HTTP form submissions using Python and cronjobs

Nobody likes doing tedious and repetitive tasks. I received an email notifying me that I had to submit a daily ‘health declaration check’ as a form on a website. Since I lead a boring and routine lifestyle, I thought I would automate this task. It would be a good excuse to refresh my knowledge on Python’s Requests module as well as get more hands on experience with cron jobs.

Using the built-in Firefox dev tools, we see that the web form consist of a pretty standard HTML form tag sent via a HTTP POST request.

Web traffic travels in requests and responses. HTTP Requests are sent out by hosts (your browser or computer) for specific web resources, which are then answered by HTTP Responses that provide the web resource requested. Requests come in multiple flavours but the most common requests are GET requests and POST requests. GET requests are HTTP requests usually sent to ‘get’ or retrieve information from the web while POST requests are are used to send or ‘post’ data to the internet (this could be adding a blog post, making a social media comment, or in our case, submitting a web form)

POST Requests are separated into 2 sections called the Head, which contain Headers, and the Body. For this particular request, the Header takes up the first 15 lines of the request and the body of our request begins on line 17.

We can see where each of the fields in the form correspond to the fields in line 17 of the request. We can thus write some code to modify the contents submittied in line 17 depending on day of the week and where I will be on that day.


Now that we have some idea of what a valid HTTP request and corresponding response looks like, we can use Python and its request module to craft and send our own custom HTTP POST request to help submit this form for us automatically.

#!/usr/bin/python3

import requests
import datetime


URL = "https://some-endpoint/somewhere/admin-ajax.php"

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0',
    'accept': 'application/json, text/javascript, */*; q=0.01',
    'accept-language': 'en-US,en;q=0.5',
    'accept-encoding': 'gzip, deflate',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'x-requested-with': 'XMLHttpRequest',
    'content-length':'661',
    'origin': 'https://some.origin.com',
    'DNT': '1',
    'Connection': 'close',
    'cookie': 'TawkConnectionTime=0',
    'sec-gpc': '1'
}

usual_places = {
    "home": "Home",
    0: "Era",
    1: "Past",
    2: "Cashflow",
    3: "Dreams-of-flight",
    4: "Stallion-Cooper-Coffee",
    5: "Ministop"
}

url_encoded_comma = '%2C'


day = datetime.datetime.today().day
if (day < 10):
	day = '0'+str(day)
else:
    day = str(day)

month = datetime.datetime.today().month
if (month < 10):
	month = '0'+str(month)
else:
    month = str(month)

home = usual_places['home']
places = usual_places[4]+url_encoded_comma+'+'+usual_places[5]


data = "action=formcraft3_form_submit&field1%5B%5D=T%E1%BB%9C+KHAI+Y+T%E1%BA%BE+H%C3%80NG+NG%C3%80Y%2F+DAILY+HEALTH+DECLARATION+FORM+2021&field19=&field3%5B%5D=Avery+Tan&field20=+&field14="+day+"%2F"+month+"%2F2021&field6%5B%5D=B%C3%ACnh+th%C6%B0%E1%BB%9Dng%2FNormal&field25%5B%5D=&field9%5B%5D="+places+"&field21="+home+"&field30%5B%5D=Kh%C3%A1c%2FOthers&field32%5B%5D=&field33%5B%5D=&field34%5B%5D=&field35%5B%5D=&field36%5B%5D=I+use+Grab&field18%5B%5D=I+agree&field28=&website=&id=51&location=https%3A%2F%2F%2F%2F&emails=&hidden=field25, field32, field33, field34, field35&redirect=&type=all&triggerIntegration=undefined"


r = requests.post(url = URL, headers = headers, data = data)


# for logging
time_now = str(datetime.datetime.now())

print(time_now + '.  Request sent. Response is ' + r.text+ '\n\n')
f = open('/home/kali/Documents/health-automated/health-logs','a+')
f.write(time_now + '.  Request sent. Response is ' + r.text+ '\n\n')
f.close()

I map the headers to a Python dictionary and also use another dictionary to record my places of interests.
I perform some logic to ensure that the date is submitted in a format that the form expects it to be.
Finally, I add these variables into my data string that will form the body of our POST request before finally sending out the request.

Now that we have our simple Python script, we now have to schedule a cronjob to automatically run the script every day.

Cron jobs are a unix time-based job scheduler where Users can set up and schedule jobs to run periodically at fixed times, dates, or intervals.

In order for a scheduled cronjob to be run at the set time, the computer needs to be running and connected to the internet during the scheduled job for it to be executed. Instead of running the cronjob from my laptop and having to ensure it is on and connected at the scheduled time of execution, I am going to use the deth* (pronounced deathstar), my raspberry pi 4 server which I have hooked into the back of a tv in my room. It’s a kali machine which I use for lightweight, in-depth, thorough scans where leaving my laptop running for 8hours is no option.

I use Dataplicity, which offers a VPN-like service to connect remotely into your Raspberry Pi without having to set up a portfoward or deal with any dynamic DNS services. Using Dataplicity, I can remotely access deth* wherever I have internet.

Cronjobs can be scheduled on a per-user basis via something called crontab. To access the user-specific crontab, we do so via

crontab -e

The syntax of a cronjob command is 5 optional parameters followed by the command you want to run at the specified time.
I’m running my cronjob at the start of the 50th minute of the 2nd hour of the day, or at 02 50 UTC time, which is about 9.50am local time.

I also redirect output from the python file to logs so that I can check periodically to make sure the cronjob is running as expected.

With this done, I can rest easy knowing that my forms are being submitted in a automated non-intervention-requiring manner!