Ad-Hoc, , balance, cleaning, variance test, data factorisation, ANOVA please follow the pdf
Page 1 of 4 INFS 2049 – UO Experimental Design Assessment 2 – Project Due: Friday in Week 10 at 5 PM Instructions: • This assessment is worth 50% of your final grade. It is due no later than 5 pm on Friday in Week 10. • You will need to submit your assignment via learnonline. The file you submit needs to be in a pdf format and prepared using the template provided. • Your submission will be marked out of 50. • The statistical analysis portion of the project is worth the majority of marks and should be well-detailed. For all analysis include the necessary assumption checking visualisations where applicable and provide thorough interpretations for the results. • All statistical tests should be conducted at the 0.05 significance level. Assessment Task Overview: Photos by Julian Hochgesang, Austin Distel and CardMapr.nl on Unsplash Digital advertising is one of the most effective ways for businesses to expand their reach, find new customers, and diversify their revenue streams. Businesses can pay to have their ads appear on popular online channels, including search engines, social media platforms, websites, and more. Digital advertising is very cost-effective since a business pays only when someone takes the desired action, clicks a link to the company’s website. An online platform will typically run an auction every time an ad space is available. Each auction decides which ads will be shown in that space. There are several ways businesses can bid for their ads, depending on what matters most to them. Most of the time, they focus on clicks, impressions, conversions or views. Imagine you are a Data Analyst and you have a client who has done some A/B testing of the bidding method they currently use, against an alternative method that has just become available to them. They would like to understand if the new bidding method has potential to bring more conversions, but otherwise do not have any direction for the analysis. You will now analyse their data and prepare a report for them with your findings. This project is your opportunity to bring together the knowledge you have acquired in this course, apply it to a real-world scenario and further develop your communication skills. Page 2 of 4 Assessment Task Details: Perform the statistical analyses indicated in this document and then write a report that explains your findings and recommendations. Follow instructions provided in this document. There are two data files for this project: control.csv and test.csv. Please refer to ‘Appendix A’ for variable descriptions. Outline of the experiment: TradeTulip.com is an online retail company that has been running digital advertising campaigns on BrandHive online platform using maximum bidding. BrandHive has recently introduced a new bidding option based on target cost. TradeTulip.com has decided to test this new feature to understand if target cost bidding would bring them more conversions than maximum bidding. Conversions are important as they allow a business to gauge the performance of their advertising campaigns. A conversion occurs when a visitor to a website completes a desired action. For TradeTulip.com, a desired action is a customer making a purchase. TradeTulip.com has set up an A/B test to compare maximum bidding to target cost bidding. The A/B test has run for one month and TradeTulip.com now expects you to analyse the results of this A/B test and report on your findings. Analysis to be performed in R: For each of the following questions, identify an appropriate analysis method (e.g. t-test, ANOVA). Do all the necessary assumption checking and perform any follow-up tests if you think they are needed or appropriate. Note: Some data cleaning and wrangling may be required. Results from your R analyses will form the basis of your report. Do not submit all your output. Instead, focus on reporting results and documenting your R code as you will need to submit it for assessment. Note: The data may not comply with some of the assumptions for the analysis method you identify for each scenario; you need to check this and consider the severity of any deviations. If you decide there is a problem, you can employ one of the following: • A data transformation (e.g. Box-Cox transformation); • A non-parametric analysis method; • Sampling to create treatment groups with equal sample sizes. 1. Begin by calculating three new measures for each date in the data set: Conversion rate = percentage of visitors to the website that make a desired action Cost per click (CPC) = Total cost of ads / Total number of clicks Cost per action (CPA) = total cost of ads / Total number of desired actions 2. Now perform some exploratory data analysis. Use summaries of the data as well as visualisations to gain insights into characteristics of the dataset and potential relationships between factors and response variables. For the report: Select some key descriptive statistics and three to four visualisations to be included in the report. 3. Is there a statistically significant difference between maximum bidding and target cost bidding based on: • Conversion rate? • Cost per click? Page 3 of 4 • Cost per action? • Cost of the campaign? • Reach? Note: Data from this experiment is paired, and you need to take this into account in your analysis. 4. Is there any other question that you think is worth asking and could be answered with the data from this A/B experiment? Pick one question and perform the corresponding analysis. Report: Once you have completed your analyses and understood your results, write a report describing your findings. Do not simply answer the questions on pages 2-3 in this document; they are provided as a guide for your analysis. Focus on interpretation and practical significance of your results. Select the most relevant output and think about how to present it effectively for your client. A template for the project report is provided in learnonline. Presentation and structure: The structure should be in a logical format that flows well. Please use the structure outlined in the report template – you can add to it with sub-headings if you wish. Since this is a report for a client, it should be presented in a professional format making it easy to read. An efficient layout is also important but do not spend too much time on making it look good and not enough time on the content. Using bullet points are OK occasionally but you will need sentences for each point (i.e. just a bullet point list with no explanation is not suitable). Word limit: There is no word limit as such, but it is expected that your report will be approximately 5 to 6 pages including relevant graphs and tables, with your R code in an appendix. Assessment Criteria: The project will be marked on how well you cover the following: Area Weighting Data analysis, incl. R code 50% Introduction and methods 10% Discussion and conclusions 25% Use of formal business or academic language, including correct grammar and spelling 10% Layout and professional presentation 5% Page 4 of 4 Appendix A – Data Dictionary Variables in the data files are as follows: Variable Description Variable name Control = maximum bidding method This bidding strategy sets the maximum amount that can be spent on a single bid. Test = target cost bidding method This bidding strategy aims to keep the average cost per action at or below the target cost at the set end date. Campaign Calendar date Date Day of the week Day The amount spent on ads in USD Spend The number of times the ad was displayed, whether it was clicked on or not. In practice, an impression occurs any time a user opens an app or website, and an ad is visible Impressions The total number of unique viewers who saw the ad. Reach The number of viewers who clicked the link to the company website displayed in the ad WebsiteClicks The number of searches performed within the company website Searches The number of times product details were viewed ContentViews The number of products that were added to the cart AddCart The number of products that were purchased Purchase INFS 2049 – UO Experimental Design Project Report Submitted by [Enter your full name here] [Enter your student ID number here] [Title of your report] Introduction The introduction should be able to be understood by a layperson and should include the motivation for the experiment as well as an outline of the contents of your report. There is no word limit. As a guideline, one paragraph will be sufficient. [Delete instruction text before submitting] [Type your introduction here] Methods Describe the experimental design, participants and variables that you have analysed. Also provide a list of analysis methods that you have used. Note: Do not include the full data dictionary; instead summarise the types of variables that were collected (e.g. psychological wellbeing). There is no word limit. As a guideline, one to two paragraphs for this section will be sufficient. [Delete instruction text before submitting] [Type your description of methods here] Results & Discussion First, summarise the main results of your analyses. You may use subsections, tables etc. as you see fit. Present and discuss results in a clear and simple way: Present findings of statistical analyses in a logical sequence. Descriptive statistics and visualisations are usually presented first, followed by the results of further analyses. Include visualisations with your results. State each result and the corresponding analysis procedure, and report P-values to three decimal places. However, do not include code or dumps of R output. Results should either be incorporated into sentences or else formatted into neatly presented tables. Next, interpret your findings by discussing their practical significance. Use plain language; there should be no technical details or statistical terminology. Are any of the results surprising in any way? Ensure you address the following: Did participants end up eating more fruit and vegetables? Which intervention proved to be effective and in what way? Were there any significant improvements to wellbeing of participants and in what sense? Finally, in another paragraph discuss shortcomings, if any, of the experimental design and analyses that were performed. There is no word limit. As a guideline, three pages will be sufficient for this section, including any tables and graphs. [Delete instruction text before submitting] [Type your results and discussion here] Recommendations & Conclusions Type your recommendations and conclusions here What do you conclude overall about the effects of increased fruit and vegetable consumption on psychological well-being of young adults, and the effectiveness of proposed app intervention? Do you have any recommendations for the client? There is no word limit. As a guideline, one paragraph will be sufficient. Do not introduce any new information in this section, and do not simply repeat statements made elsewhere in your report! [Delete instruction text before submitting] [Type your recommendations and conclusions here] Appendix Your full R code goes here, including data prep and all relevant condition checking. Make sure that the code is well commented. In particular, clearly indicate which questions are being addressed with each section of the code, and what is to be achieved. Comment briefly on the outcome. Do not include the full output from your code. [Delete instruction text before submitting] 1