CSE115 Project - Part 4
Front-end and Web server requirements
Submit a ZIP file of your repl in AutoLab as Project Part 4 by Fri. Dec. 3 at 5:00 PM.
This submission is worth 48 out of the project's 60 total points.
NOTE: You have a limited number (5) of AutoLab submissions. Use them wisely. Your LAST submission counts.
Languages: JavaScript and Python
Topics: lists, dictionaries, loops, conditionals, reading CSV files, writing CSV files, HTML, JavaScript front end, Bottle web server, JSON, AJAX (lectures up to and including April 9)
Overview
In this last part of the project you will put the pieces from earlier parts of the project to create a fully functional web application. There are several pieces you need to complete; each is relatively small, but you need to work incrementally to ensure that all the pieces work together in the finished product.
If you did not quite complete the functions for parts 1, 2, and/or 3 keep working on them. Remember that the bulk of the points come from your final project submission.
Your finished web application must be able to create three graphs. Two of these graphs will be shown when loading the page; the third graph is created on demand. Because the CDC's vaccine data is updated daily, the data your program downloads (and is then shown in your graph) will change with time. Because I need to prepare this assignment well ahead of time, your graphs will not be identical to those shown here (but should be similar). You can use AutoLab to check your code for the first 3 parts. Downloading the CSV file your program saves and then opening it in a spreadsheet application can help you verify your results.
The first graph you will need is a
bar chart
showing the most up-to-date results for the percentage of a location's population that has been fully vaccinated:
The second graph is a
pie chart
showing the number of administered vaccines based upon the vaccines manufacturer. As with the first graph, this should use the most recent data available. While the data will be different, the graph should look something like this:
The third graph is a
line graph
showing the percentage of a location's population that has been fully vaccinated (e.g., the values associated with 'series_complete_pop_pct') over time. Unlike the first two graphs, this graph's data will be limited to a specific location and so will not be generated when the page starts up, but is instead generated whenever the user clicks a button. Use the strings associated with the 'date' key for your x values; as the image below shows, Plot.ly is able to interpret them as dates:
Web server (Python)
Define your Bottle web server so it has five routes:
The "/" must serve up the HTML file as a static file.
You may use any name you want for the remaining routes. Best practices encourage selecting route names that are meaningful and descriptive within the context of the application.
A route to serve up the JavaScript code file as a static file
A route to serve up the bar chart data as a JSON blob
A route to serve up the line graph data as a JSON blob
A route to serve up the pie chart data a JSON blob
HTML file
You will need to write an HTML file for your application. Inside your HTML file's element, it will need at least 2 elements to load in the needed JavaScript files. The first element should cause the Plot.ly library to be loaded. The URL for this is:
https://cdn.plot.ly/plotly-latest.min.js
You will need at least one more element to load in the JavaScript code you must supply. You can follow the class example of using multiple files OR just put all the code in one file. Since the file(s) are on the same web server as your HTML file, the elements' src property should use the route specified in your web server (i.e., do not include a protocol or server).
Your HTML file also needs a element. The element will need to include an onload property. This should call a function in your JavaScript code which sends 2 GET requests to the web server: one for the bar chart and one for the pie chart. When the server responds to each request, you will need code that creates the graph using that data. As an example, the element in my code is:
Inside the element, you will need to include 3 elements. Each element will need to include an id property and be assigned a unique value. You will need to use these ids to display your graphs.
You also need a textbox and button on your page. These should be located in the file just before the line graph's element. The textbox needs to include an id property and be assigned a unique value for this id. The button will need to include an onclick property. This should call a function in your JavaScript code which gets the value of the textbox and sends that to the server using a POST request. When the server responds to this request, you will need code that creates the graph using that data. As an example, the textbox and button elements in my code are:
Location:
Plot it!
Data processing (Python)
You will also need additional code to process the data. The functions required for parts 1, 2, and 3 of the project will make this process much simpler. You should make certain they are available for this part of the project (copying them in as needed).
Graph #1 - the bar chart
This chart must show the percentage of each location's population that has been fully vaccinated. After reading the dataset from your CSV file, find the maximum value associated with the 'date' key. By only using the dictionaries with that date, you can generate the most up-to-date results.
Your JavaScript code will need the values associated with the 'location' key (these are used as the x values) and the values associated with the 'series_complete_pop_pct' key (these are used as the y values). There are no requirements for the specifics of how to provide this data in your AJAX response. You should use the structure that you find easiest.
Graph #2 - the pie chart
This chart must show how much of each manufacturer's vaccine has been administered. After reading the dataset from your CSV file, find the maximum value associated with the 'date' key. By only using the dictionaries with that date, you can generate the most up-to-date results. To find the number of vaccines administered you will need to find the sums of the values of each of the 'administered_janssen', 'administered_moderna', 'administered_pfizer', and 'administered_unk_manuf' keys.
Your JavaScript code will use the strings 'Janssen', 'Moderna', 'Pfizer', and 'Other' for the labels and the sums of administered vaccines for each manufacturer (these are used as the values). There are no requirements that all of this data be in your AJAX response nor any specifics for how to respond to the AJAX request. You should use the structure that you find easiest.
Graph #3 - the line graph
This graph will be generated for the location which is specified in the POST request. For that location, the graph shows the percentage of the population that has been fully vaccinated over time. After reading the dataset from your CSV file, generate a list of the dictionaries whose location is equivalent to the one sent in the POST request. You should then sort that list using the values associated with each dictionary's 'date' key. (Because of how the dates are written, sorting them alphabetically will also sort them chronologically).
Your JavaScript code will need the values associated with the 'date' key (these are used as the x values) and the values associated with the 'series_complete_pop_pct' key (these are used as the y values) There are no requirements for the specifics of how to provide this data in your AJAX response. You should use the structure that you find easiest.
Fetching and caching CDC data
The following function uses your json_loader and save_data functions to get data from the CDC site. In doing this, it includes two additional optimizations. The function begins by checking if a file named saved_data.csv already exists. When the file exists, it will not request data from CDC and rely on the data you previously downloaded. This minimizes the likelihood of your application going over any resource limits and being prevented from making CDC requests. The query further reduces overheads using the
query strings defined by the CSC API. While these query strings are specific to this API, this API is used by a
large
number
of
governmental
data sites.
You should include this in your code and call this function just before starting your webserver.
import os.path
import data # This assumes that your functions from parts 2 & 3 are in a file named data.py
def load_data( ):
csv_file = 'saved_data.csv'
if not os.path.isfile(csv_file):
url = 'https://data.cdc.gov/resource/unsk-b7fc.json?$limit=50000&$where=location!=%27US%27'
info = data.json_loader(url)
heads = ['date','location','administered_janssen','administered_moderna','administered_pfizer',\
'administered_unk_manuf','series_complete_pop_pct']
data.save_data(heads, info, 'saved_data.csv')
JSON and AJAX
JavaScript front-end and Python back-end run on different machines. (JavaScript is executed by the machine of the person looking at the page; the Python code is executed by repl.it's servers.) This means that we cannot communicate directly from JavaScript to Python (or vice versa). We instead rely on an approach called AJAX to allow the two sides to coordinate their work.
This process begins when the JavaScript front-end needs data from the Python back-end. To make this request, it must do two things:
Send a request to the repl.it server requesting the data;
Tell the JavaScript system what function should be called if/when it receives this response.
GET Requests
When you are not sending data in the request, you need the ajaxGetRequest function. We will discuss how to use this function during lectures, but you will need to copy-and-paste this code to use it:
// path -- string specifying URL to which data request is sent
// callback -- function called by JavaScript when response is received
function ajaxGetRequest(path, callback) {
let request = new XMLHttpRequest();
request.onreadystatechange = function() {
if (this.readyState===4 && this.status ===200) {
callback(this.response);
}
}
request.open("GET", path);
request.send();
}
You will need to provide two arguments for each of your 2 (or more) calls to ajaxGetRequest. An example of this call is:
ajaxGetRequest("/vacByLoc", showBar);
The first argument ("/vacByLoc") is the route on the repl.it server where the request is sent. For this to work, the Python back-end will need a function with a matching annotation (e.g., @bottle.route("/vacByLoc")). The second argument (showBar) must be the name of the JavaScript function you wrote. JavaScript automatically executes the second argument (showBar), but only when it receives a response from repl.it. That function should include one parameter. This parameter will be the data the back-end sent over the Internet to respond to our request. As an example, the first bit of my function looks like this:
function showBar(response) {
// Data is always sent in JSON, so we must first convert it
let data = JSON.parse(response);
// Now write the code that uses data to create the scatter plot
Where do we make the calls to the ajaxGetRequest function? For this project, these should be in the JavaScript function called in the tag's onload property. In
the example above
the function getLocData would include the call to ajaxPostRequest. In
the example above
the function getData would have all of the calls to ajaxGetRequest.
POST Requests
When we are sending data along with the request, you will need to use the ajaxPostRequest function. We will discuss how to use this function during lectures, but the code you should copy-and-paste is:
// path -- string specifying URL to which data request is sent
// data -- JSON blob being sent to the server
// callback -- function called by JavaScript when response is received
function ajaxPostRequest(path, data, callback) {
let request = new XMLHttpRequest();
request.onreadystatechange = function() {
if (this.readyState===4 && this.status ===200) {
callback(this.response);
}
}
request.open("POST", path);
request.send(data);
}
You will need to provide three arguments for your 1 (or more) calls to ajaxPostRequest. An example of this call is:
ajaxPostRequest("/vacByDate", locBlob, showLine);
The first argument ("/vacByDate") is the route on the repl.it server where the request is sent. For this to work, the Python back-end will need a function with a matching annotation (e.g., @bottle.post("/vacByDate")). The final argument (showLine) must be the name of the JavaScript function you wrote. The final argument must be defined just like the
callback example in the GET request. Also like that earlier example, JavaScript will automatically call the callback function in a POST request upon receiving the server's response.
Where do we make the call to the ajaxPostRequest function? For this project, this would be in the JavaScript function called in the tag's onclick property. In
the example above
the function getLocData would include the call to ajaxPostRequest.
GRADING
For your final project submission, autogrades in AutoLab will score the functions specified in parts 1, 2, and 3. The functionality for part 4 of the final project will be manual graded by the TAs in the week after the project deadline. As before, you are limited to 5 submissions on AutoLab and it is your last submission that counts: that is the autograder score we will use and the submission that will be manually graded by the TAs.
SUBMISSION INSTRUCTIONS & RESULTS
In order for us to grade your project, you must submit all of your .html, .js, and .py files. We will download and execute your project for the manual grading, so make certain everything runs before you submit.
Code organization for submission
You must follow these instructions specifying the file where each function is required. They are for your submission to be able to earn points, since the autograders will be relying on this.
Your Python code MUST be separated into 3 files and the files MUST have the following names:
data.py - this file must contain ALL OF THE FUNCTIONS FROM PARTS 2 and 3. Do not include any web server code and the file cannot import the bottle library nor can it use the bottle library.
processing.py - this file must contain ALL OF THE FUNCTIONS FROM PART 1. You can include the functions from part 4 that process the data, but this is optional. Do not include any web server code and the file cannot import the bottle library nor can it use the bottle library.
main.py - this file should include all of the Python code that defines the web server's behavior. This file should have the following basic structure:
import bottle
import json
import data
import processing
# ...all the functions receiving the AJAX requests defined here...
import os.path
def load_data( ):
# You should use
my code for load_data()
load_data()
bottle.run(...)
Note the import data and import processing lines. You will need to include the name of the module when you call functions that are in those files. For example, when load_data wants to call the json_loader function, it writes the call as data.json_loader(url).
Feedback
Autolab will give two kinds of feedback:
An ungraded checklist showing how well your code meets (some of) the explicitly stated expectations spelled out above in this document. The first bit of the feedback will look something like this:
******************************************************************************
*** File names and quantities ***
******************************************************************************
PASS: I expected to find 1 HTML file. I found 1:
index.html
PASS: I expected to find at least 1 .js file. I found 1:
vaccine.js
PASS: I expected to find 1 file named 'main.py'. I found 1:
main.py
PASS: I expected to find 1 file named 'data.py'. I found 1:
data.py
PASS: I expected to find 1 file named processing.py'. I found 1:
processing.py
Please verify that all of the checks are labeled "PASS". These automated tests are not graded, but will help us make the manual grading go much faster. Small mistakes are not a problem (such as if you add spaces where we did not expect it), but you should verify any test labeled FAIL is just a difference in spacing. We gave you lots of leeway for how to structure your code for part 4, but may not be able to find and grade code if you move things around too much.
A report on the automated grading of the functionality of parts 1, 2, and 3. Each part is graded out of 10 points.