Last updated on June 11th, 2023 at 09:34 pm

The paper ” How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection ” 　published by researchers belonging to Shanghai University of Finance and Economics in China describes the characteristics and differences between the content answered by ChatGPT and human experts. This is a research report that investigated the

Since its release in November 2022, OpenAI’s ChatGPT has attracted attention in many fields such as academia and industry, including the natural language processing (NLP) community. In terms of usability, it far surpasses conventional public chatbots and is being used in a variety of places.

On the other hand, there are cases where ChatGPT is used in bad ways, such as for papers and homework, and for cyberattacks, and some groups have banned ChatGPT. In addition, ChatGPT’s response is limited, and there are cases where inappropriate responses are returned.

This study investigates and analyzes a large number of questions and answers to the question of what is the difference between a human expert and ChatGPT. For the analysis, we collected about 40,000 questions and corresponding human/ChatGPT answers (in English and Chinese). It covers a wide range of domains (computer science, finance, medicine, law, psychology, etc.) and is named HC3 (Human ChatGPT Comparison Corpus) dataset.

In order to investigate the characteristics of ChatGPT’s answers and the differences between them and human experts, we first compared the content generated by ChatGPT with that of humans, and performed a comprehensive evaluation by humans. Specifically, we asked all participants to look at and compare both answers and summarize the characteristics of each. We ended up with over 200 feedbacks.

Read also: A.I Content Detector: 5 Best AI content Detection Tools

Difference Between ChatGPT And Humans

　Below is a characteristic pattern of ChatGPT.

• ChatGPT writes in a clear logical, organized manner, and likes to define the core concepts of the problem without sacrificing generality.
• ChatGPT tends to give long and detailed answers unless you give a prompt like “please explain in one sentence”.
• ChatGPT shows little prejudice or harmful information, is neutral on sensitive topics, and has little stance on political spheres and discriminatory harmful conversations.
• ChatGPT will not be able to answer inquiries that require information after September 2021, and will tend to refuse to answer if it lacks knowledge.
• ChatGPT may fabricate facts. For example, for legal questions, ChatGPT may fabricate non-existent legal provisions to answer the question. Also, if you ask a question for which there is no existing answer, ChatGPT may fabricate facts to provide an answer.

　Below are the main differences between humans and ChatGPT.

• ChatGPT answers are strictly focused on the question, whereas human answers are divergent and easy to shift to other topics. Humans are more divergent in terms of content richness, but ChatGPT prefers to focus on the question itself.
• ChatGPT gives objective answers, but humans tend to prefer subjective expressions. ChatGPT generates safer, more balanced, neutral and informative text than humans. As a result, while ChatGPT is good at interpreting technical terms and concepts, human responses are more specific and based on legal provisions, books, and articles, especially when making suggestions for medical, legal, and technical issues. It tends to include detailed citations of the source.
• ChatGPT responses are usually formal, whereas human responses are more colloquial (spoken). Humans tend to be more concise, with more abbreviations and slang. Also, while humans like to use humor, sarcasm, metaphors, and analogies, ChatGPT never uses irony.
• ChatGPT doesn’t express much emotion, but humans choose many punctuation marks and grammatical features depending on the context to convey their emotions. Humans use multiple exclamation marks, question marks, and ellipses to express strong emotions, and various parentheses to explain things. ChatGPT, on the other hand, prefers to use conjunctions and adverbs such as “generally”, “on the other hand”, “first, then, finally” to convey a logical train of thought.

Statistical analysis results

Next, we statistically analyze the lexical features of the collected corpus. First, we compared the number of words used in the responses and the density of how many different words were used in the sentences. For example, even if you write an article of 1000 characters, if only 100 kinds of words are used, the density is low.

Conversely, even if the sentences are the same length, if many different words are used, the density will be high. Analysis results show that human responses are relatively short compared to ChatGPT, but that more vocabulary is used. This indicates a higher density in humans than in ChatGPT.

Next, as a result of part-of-speech analysis, ChatGPT uses many nouns (NOUN), verbs (VERB), determiners (DET), adjectives (ADJ), auxiliary verbs (AUX), conjunctions (CCONJ), and particles (PART). However, I found that adverbs (ADV) and punctuation marks (PUNCT) are not often used. A high proportion of nouns tends to make the text more argumentative, indicating informationally and objectivity.

Along with this, the frequency of appearance of pronouns (ADP) and adjectives (ADJ) tends to increase. The frequent co-occurrence of the conjunction (CCONJ) indicates that the article structure, causal relationship, progression relationship, and contrast relationship are clear. They are also typical features in scholarly articles and official documents.

Next, in order to investigate how “emotional” ChatGPT is compared to humans, we use a multilingual sentiment classification model fine-tuned on the Twitter corpus to perform sentiment analysis on comparative data in English and Chinese. carried out.

The results show that both humans and ChatGPT have the highest percentage of neutral emotions, with ChatGPT expressing more neutral emotions than humans. Humans also express significantly more negative emotions than ChatGPT. The percentage of human positive emotions is slightly higher than ChatGPT. Overall, ChatGPT is less emotional than humans, but not completely emotionless.

Another analysis found that humans have the ability to express themselves in different ways, depending on the context, audience, and purpose of the writing they write. This includes using creative or imaginative elements such as metaphors, metaphors, and unique word choices.

Source and Image Credits: Guo, Biyang, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue and Yupeng Wu. “How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.” (2023) ).