Introduction

The purpose of this webpage is to show how efficient and accurate ChatGPT is, using questions from social sciences, history and computer security, we asked ChatGPT 150 questions, and tested ChatGPT's accuracy with the result given.

Profiles

• Zijing Wang

  SDE Intern @ Tecsys, Prev SDE Co-op @ RBC, CIBC

  JavaScript, TypeScript, React, Node.JS, Express.JS, MongoDB, Mongoose

• Jinxiao Zhou

  4th year student from York university

  HTML, CSS, JavaScript, GitHub

• Michelle McManaman

  Senior at York University

  HTML, CSS, JavaScript, Node.js, Express.js

ChatGPT Efficiency / What we are working with

• Research Approach

  • Database setup: We used cloud MongoDB to collect dataset for each domain, which is more accessible in this case compared to SQL. Then populated each collection with 50 questions provided, then used the anticipated answers to compare with ChatGPT’s answers.
  • Server setup: Utilized Mongoose ODM. It provides existing methods and functions for Node.js. Used Node.js and Express.js for backend servers to manage database and API interactions.
  • Implemented Nodemon app.js that helped developed Node.js based applications by automatically restarting the node application when file changes in the directory are detected.
  • API integration: as per requeste, we used ChatGPT’s API to process the questions, returning responses are stored in the cloud MongoDB as well.
  • Testing: at last, we tested locally to ensure we entered the correct data and codes for fetching and storing data.

• Dataset Overview

• The dataset we used for testing includes 3 field of studies: history, social science, and computer security.
• With History questions, its nature is primarily analytical and descriptive, focusing past events and their significance. Answering them requires contextual understanding of timelines which emphasizes critical thinking and synthesis, even so, answers are quite similar since it's past events.
• With Social science questions, its nature is primarily critical and theoretical, it focuses on societies and interactions, answers may vary depending on each perspective or context, it encourages critical reflection and evaluation, which often leads to debates or case studies, that may cause different answers.
• With Computer security questions, its nature is technical and problem-solving, focusing on protecting systems, networks and data from different types of threat, when answering, it requires precise understanding of the situation, analyzing risks and real-world experiences when solving these questions, thus answering these question may vary because of the questions and situations it describes.

---

• Results

  Run Test

  Domain	Avg Response Time	Min Response Time	Max Response Time	Accurate Rate
  Histroy	510.07ms	277ms	654ms	75%
  Social Science	548.03ms	364ms	636ms	80%
  Computer Security	495.71ms	256ms	612ms	78.3%

Evaluation and Visualization

Data Analysis & Key Findings

• Performance across different domains & Trends: as we can see from the results. Questions in social science take the most average time to respond, then computer security, then history; while accuracy rate goes in social science, computer security, then history. We can see a small trend that the slightly longer the response time is, the slightly more accurate it would be.
• Limitation: since it does costs money, we were able to conduct our analysis using version 3.5 of ChatGPT. We think if we were to use a better version (for example version 4.0), we might have better results and accuracy towards the average response time. Another limitation we have is for our demo above, we used 5 questions for the remote protocol call.

Concluding Remarks

• What we learnt:

• For Future improvements we thought about using different APIs to test the efficiency and accuracy of questions and answers.