EVALUATING THE TRUSTWORTHINESS OF ONLINE SOURCES: ASSESSMENT FOR STUDENTS OF UNIVERSITAS INDONESIA

In 2019’s presidential election in Indonesia, the Internet was flooded by unverified and contradicting claims, which only blurred the lines between fact, opinion, and straight-out hoax. This research investigates how an educated group (college students) evaluates online sources in the context of Indonesia’s political campaigns. An assessment was designed for the students to judge the credibility of five political news articles appearing online in April 2019, the election month. The framework used was inspired by similar assessments conducted by the Stanford History Education Group, researchers who study digital literacy among American students. For this research, fifty students from five faculties in Universitas Indonesia identified false information on the Internet and explained how they recognized it. Their explanations were then classified into three: Mastery, Emerging, and Beginning. The results show that, by and large, the students could recognize sources containing false information. However, only a small percentage provided logical reasoning on Mastery level. It is hoped that the findings give insight into how the students assess online sources and complement other similar works that investigate hoax and political campaigns in other parts of the world.


INTRODUCTION
The Internet is our friend and foe, all at the same time. While it opens the door to knowledge and endless possibilities, it is also laden with hidden threats for those who do not tread with caution. The research by the Stanford History Study Group (SHEG) shows how dangerous the Internet can be. Through years of extensive research, they found that even the most educated Americans might still not know how to safely navigate the Internet, especially when identifying false information (Wineburg, S., McGrew, S., Breakstone, J. & Ortega, T., 2016). Since 2014, SHEG has developed short assessments that require their respondents-American students from middle school to university level-to evaluate social and political information they find on the Internet. The team calls the skills needed to assess these online sources Civic Online Reasoning (COR). The COR assessment adjusts the educational level. For example, middle schoolers distinguish between a news story or an advertisement, whereas college students choose credible research papers. In distributing the assessments, the research group does not discriminate. SHEG tests students from both reputable schools and those that are under-resourced. In 2016, the group released a report showing that the overall results were not encouraging. Regardless of their various backgrounds, they made poor judgments about what to trust on the Internet.
In Indonesia, both teachers and students also rely on the Internet to support their teaching and learning experiences. Nevertheless, the threat remains the same: without proper precautions, the Internet brings unforeseen consequences. One skill needed to safeguard these web surfers is called digital literacy. It means that one has "the ability to find, evaluate, utilize, share, and create content using information technologies and the Internet" (Cornell University, 2009). On the other hand, Gilster (1997) thinks that the most important skill is making educated judgments when surfing the Internet. Therefore, digital literacy, in line with COR, also needs to be taught and learned. Indonesia's current education system has not explicitly ruled digital literacy in the curriculum, but the need for awareness has been acknowledged. Although the notion of digital literacy is still peripheral to the teaching of subjects matters taught in schools and campuses, more schools have taught students how to be more critical of the texts they receive. Students have also been exposed to the public and governmental campaigns on digital literacy (Panjaitan & Mahesa, 2019). Therefore, Indonesian students are expected to be more critical of online texts. Similar to what the SHEG group has done, this research investigates how our educated group, or the Indonesian students, criticize texts available online. The texts chosen are about the Indonesian presidential race in 2019, relevant to the political situation that generates many false claims circulating the Internet.
There have been papers written about false information and political events in Indonesia. Using a memetic practice approach, Utami (2018) analyzed fake news surrounding Jakarta's gubernatorial elections that drew the rest of Indonesia's attention. The 2017's election was controversial since it involved a candidate from a minority group: a Chinese Christian man named Ahok. Hoaxes that surfaced around the election were mostly about his religion and race, and these hoaxes were rampant because they could easily affect people's sentiments. Memes in Utami's work described the transmission of ideas through the Internet, and hoax is perceived as memes that have the power to influence people's minds into believing false information. The memes work by amplifying stereotypes and beliefs in society and presenting them as part of society. Consequently, "a hoax has the potential to be a means of political partisanship" (Utami, 2018, p. 85).
Two years after the gubernatorial election in Jakarta and Ahok lost the race, Indonesia still faced the same problem. Hoaxes in political campaigns remained a problem in Indonesia's political setting. The 2019's presidential election was accompanied by rumors about the two candidates, the incumbent President Joko Widodo and his opponent Prabowo. The rumors appeared online and quickly spread through personal messages. According to Prayogo (2018), these false rumors were black campaigns that mainly aimed to reduce the candidates' credibility. Tukina and Sanjaya (2020) stated that the media helped spread misinformation relating to the general election, and among the many issues that were circulated were about religion and race. Joko Widodo's mother was falsely reported as Christian, and that Joko Widodo was of Chinese descendent. Prabowo, being a Muslim, was not free from the religion-based black campaign. The issue focused on Prabowo's extended family that is a mix of Muslims and Christians.
There have been many attempts and proposed solutions to stopping the spread of hoaxes. Rashkin, Choi, Jang, Volkova, and Choi (2017) researched the language of fake news and how the research could help with automated fact-checking and fake news detection. Their research tried to find the linguistic characteristics of fake news that are not found in verifiable news. Tukina and Sanjaya (2020) also offered solutions to the problem, which called for participation from the media to write only verified news. Like Rashkin, et al. (2017), they also saw the importance of clarifying a misleading claim and the need to have a reliable fact-checking mechanism.
Nonetheless, the media is not the only one responsible for controlling the spread of hoaxes, and corrections are not always practical. A study by Flynn and Krupnikov (2018) suggested that debunking a hoax seldom works. They based their argument on psychological research about motivated reasoning. Motivated reasoning explains why people do not easily accept correction that does not favor their preferences and beliefs. People are motivated to ignore positive information about a person they do not like and prefer to hold on to negative information. Therefore, busting hoaxes based on the third party's finding (fact-checkers) is arguably not the most effective solution as people will only find justification to dislike a person. False information is a recurrent phenomenon where misinformation persists. The cases stated above show that hoaxes are easily generated and spread. What needs to be nurtured is more active participation from the Internet users. We need them to be aware of this threat and be more critical of the online texts they receive and think before forwarding them. One way to foster this good habit is by promoting digital literacy. that the students assess an online text, and they have to decide whether it is credible or not and why. Their reasoning is then classified into three levels, which are Beginning, Emerging, and Mastery. This paper also uses the three classifications, with modifications to better fit this research's objective (Table 1). Table 1 The Rubric

Mastery
Student has correctly evaluated the item and provides coherent reasoning. Emerging Student has provided a correct evaluation, but provides limited, incomplete, or incoherent reasoning. Beginning Student incorrectly identifies the item as trustworthy or false. Student uses weak or incorrect reasoning.
The assessment designed for this research underwent significant adjustments from those made by SHEG. For this research, the assessment focused on five news articles that might contain false information regarding the election. Out of the five articles evaluated, one was different. The article labeled Data 1 was a credible source. The National Press Board verifies the news portal, and it also reports valid information following journalism ethics and standards. However, the headline was quite misleading, using the hyperbolic word "exodus" that was later proven rebuked by the article's content itself. Although clickbait is one indicator of fake news, the headline was more to false advertising for page view than false information. A good reader needs to evaluate a source's credibility from more than just an angle to determine the validity of a source-that is, to read beyond the heading and the content. Therefore, the assessment for this research was divided into two. Data 1 was to see the students' logical reasoning. In contrast, the other four data, Data 2 through 4, examined their general ability to identify false news from their visual indicators.
The five data were collected and chosen based on the Ministry of Communication and Information Technology Indonesia or Kominfo (Zhacky, 2019), which saw a hoax spike during April 2019, the highest the election in the same month. Out of the 22 hoaxes about the election, five were chosen as the data, as shown in Table 2.  *That there was a massive outbound number of Indonesian citizens before the election for fear of chaos was indeed one of the hoaxes listed by Kominfo. However, suara.com clarifies this issue in the article presented as Data 1.
The assessment was distributed online to five different faculties in Universitas Indonesia, with ten respondents from each faculty. Three faculties study social and humanities subjects (Faculty of Humanities, Faculty of Social Sciences and Politics, and Faculty of Law), whereas the two other groups study hard sciences (Faculty of Engineering and Faculty of Mathematics and Natural Sciences). The assessment was given to the third-semester students or higher than that. They should have learned about critical thinking in at least one of their subjects in the first two semesters (i.e., MPK Terintegrasi or Integrated Personality Development Skills). After they agreed to continue with the assessment, they were required to fill out questions regarding their personal and political background, which we granted secrecy. They knew that the test had no time restriction, and they were allowed to do independent research before deciding on their answers. The pool of respondents is as follows.
The percentage of students studying in semester 3, 5, and 7 was shared almost proportionally, at around 30 percent each. Only a small fraction of the respondents from semester 9 took the test, and this group was the hardest to collect since they mostly graduated after the seventh semester.  Another thing about the respondents is that they mostly supported the incumbent Jokowi and his partner Ma'ruf Amin. Almost three-quarters of respondents (37 students) chose Jokowi, and the rest of the pie was almost equally shared by Prabowo supporters (6 students) and those who remain abstain or "golput" (7 students).

DIGITALLY LITERATE, BUT NOT YET ON MASTERY LEVEL
From the gathered responses, the students were generally quick to dismiss questionable news sources but not quite there in terms of logical reasoning. Data 1 was the most difficult to determine out of the five news sources, as shown by the almost half-and-half answers received (Fig. 4).

Fig.4 Data 1 Response
Many respondents seemed to rest on knowing that the news portal was credible enough. However, they neglected the fact that the headline was inconsistent with the actual content. While the headline stated that there had been an "exodus" before the general elections, the clarification given by the Coordinating Minister for Political, Legal, and Security Affairs Wiranto proved otherwise. Only a small percentage of the respondents looked beyond the headline and checked for more convincing external sources. Another small percentage did not go further than the website but was thorough enough to identify the headline's nature, which was more of a clickbait. The subheading, just below the picture and before the news, clarified that the number of in-bound citizens surpassed the outbound number by 4,000. These Mastery-level respondents did a lateral reading, checking for facts by going to other online sources (Wineburg & Mcgrew, 2017). Fact-checkers use this kind of reading, and by so doing, they avoid the trap of seemingly convincing sources.

Table 2
Samples of Mastery Level
Unfortunately, only 18 percent of the respondents admitted to reading laterally. The biggest portion grounded their evaluation based on what they think was enough to label a source "valid", as seen in Table 3.  Mencantumkan tanggal publikasi berita dan nama pembuat berita.
An almost a-third portion of the respondents resorted to website reliability (28%). While this type of reasoning is a step to being digitally literate, there is a potential problem that needs to be addressed. Determining website credibility can be a problem of its own, especially with the many news portals emerging on the Internet. One way of checking this credibility is by going to the National Press Board (Dewan Pers) website to check if a news portal is verified or not. Although being a verified news portal is not a sure-fire way to guarantee the news articles' objectivity, the news portal has somewhat credibility to maintain. It also helps readers filtering out news portals that are increasing by number every passing day.
Apart from the website reliability, there is another problem lurking if readers do not go beyond reading the news article presented. As many as 12 respondents (24%) judged a news article's credibility from the news source quoted by the journalist. In Data 1, the person quoted was Wiranto, the Coordinating Minister for Political, Legal, and Security Affairs. The problem was that false news also used false names and organizations to convince the readers, as later will be seen in Data 2, 4, and 5. Wiranto did give the right statistics quoted in Data 1, as verified in other credible news outlets. However, relying on the news source without checking further can lead to being duped online. Another point that may give loopholes to false information is their journalism ethics and standards. As many as eight respondents (16%) thought that by being given the date, location, author, and editor names, they could determine news's validity. Whereas a good news portal will provide such information, false news can also mimic the same gesture to convince the readers. Therefore, this group of respondents had been quite observant in reading a text, but their logical reasoning still lacked a more comprehensive measure (Emerging). The rest of the respondents (14%) sadly still fell under the Beginning Level, as seen in Table 4. This group shows incoherent reasoning. The respondents might not go beyond the heading to find that it was clickbait, and they falsely judged that the website was not credible. Others did not give explanations that could justify their verdict (incoherent). One important thing to highlight is that a video is not always proof of authenticity. Video manipulation is one way of false news to gather trust (Panjaitan & Mahesa, 2019). Expecting evidence in the form of a video to prove credibility is not only insufficient but can also be unfitting. Table 5 gives a summary of the respondents' reasoning.

IDENTIFYING FAKE NEWS: HITS AND MISSES
This section, or the second part of the analysis, focuses on four non-credible news articles labeled Data 2, Data 3, Data 4, and Data 5. The four remaining data had some evident and lessthan-obvious indicators of questionable online sources that the respondents should identify. Most respondents were aware of some obvious signs of a questionable news source. Among the apparent signs was the word "viral" in the headline, found in Data 3. This catchphrase has been known to be found in many false news articles, encouraging susceptible readers to accept it as a common phenomenon (Panjaitan & Mahesa, 2019). Only fewer than 10 percent of respondents neglected this "viral" indicator. Even though the news content was not automatically false due to the word "viral", several other indicators should be further inspected from Data 3. Data 3 did not include any external source to support this report, making it lack the necessary indicators for a credible source. The other 90 percent noticed this and, apart from that, identified that writing was more like a social media/blog post designed to look like a news portal. Indeed, upon further inspection, the so-called news articles posted on the "news portal" contained biased articles that only aimed at defaming political figures that were rivals of Jokowi.
Biased reporting was also found in Data 2, where the portal dan article supported Prabowo. The inclination should have been identifiable from the hashtags positioned at the bottom of the article (e.g., #BPN Prabowo, #Jurkam BPN Prabowo -Sandiaga, #Prabowo menang). However, almost all respondents were not careful enough to identify the hashtags. Data 2 showed this recklessness, in which no-one mentioned the hashtags aimed at exalting Prabowo's win before the official announcement. Thirty percent of the respondents were led to believe in the news since the article "quoted" names from the national boards like The Election Supervisory Board (Bawaslu) and The General Elections Commission (KPU) that seemed to support the claim. Only a few respondents noticed the news' logical fallacy, stating that the sources quoted did not have the authority to determine a case like this (the dispute). The sources quoted in the news carried prominence in the respondents' logical reasoning. In many responses, they showed immediate trust when the news sources were put in quotes. To them, this proved the news' credibility. When no figures were quoted, they stated their distrust in the news (see Table 5 about news source). Ironically enough, this is used as a loophole by the newsmakers. News that contains false information foregrounds this by including quotes that are not verified and often a complete mockery to their readers' knowledge. Data 5 is an example of this. The news reported that the Secretary-General of the United Nations has officially congratulated Prabowo as the elected winner. The name "Pedro Coelho" mentioned in the news was not the Secretary-General. Upon a simple Internet search, it should have been known that António Guterres was the presiding Secretary-General. Those who were aware of this incorrect naming quickly identified the news as a hoax. Unfortunately for a fraction of respondents, they fell right into the trap.
Another challenge is the trap of the algorithm. Data 4 disinformed readers by quoting a misleading source, but the power lay in the number of other articles, also quoting the same source. The SOHN Conference Foundation is an existing international organization that is far from dealing with political causes. However, an amateur google search returned 168,000 results when the keywords SOHN and general election (pemilu) were typed together. The number also included a rebuke on the so-called research from seword (Rohindo, 2019), a famous pro-Jokowi news portal, and some other verified news portals like detik, liputan6, and CNN Indonesia that gave opposing results. However, out of the total 9 (nine) results appearing on the search's first page, four noncredible sources supported the false claim: www.law-justice.co, www.kabartoday.co.id (Data 4), www.suarakarya.id, and go-riau.com, with the first three appearing on the top results. Without further inspection of the data included in a news article, it would be easy to fall prey to nontrustworthy sources online. Figure 5 summarizes the percentage of correct and incorrect evaluations of Data 2, 3, 4, and 5.

DISINFORMATION AND ELECTION, A GLOBAL TREND
The data collected may have been very segmented and did not cover a more diverse participant group. However small the scope was, we must not forget to see that what the research shows reflects a more significant issue at hand: disinformation, digital literacy, and socio-political practices in Indonesia and the world. In 2016, the phrase "post-truth" claimed the spotlight as Trump raced to the US presidency (Word of the Year 2016 is…, 2019). Although he and his campaign team were widely criticized for offering the alternative truth (opinion and false claims presented as truth), America inaugurated him president in January 2017. Trump's victory may have shocked people, but the practice was not a singular phenomenon. In India, the national election was marred by disinformation coming from both the incumbent and the opponent. Both parties went to social media platforms like Facebook and Whatsapp, which generated a massive campaign fueled by rumors. The phenomenon was described as the "Whatsapp Elections" (Hui &  Prakash, 2019). In the context of the European Union (EU) election, particularly concerning the topic of Brexit, disinformation spread in Facebook groups. With catchphrases like "Make Britain Great Again" and by groups claiming to be "Libertarians and Chartists for Trump", false stories were fabricated around the issue of Brexit, immigration, and France's yellow vest movement (Spring & Webster, 2019). A more imposing question to be asked would be why people could fall easily to ridiculous schemes. A pollster Indikator Politik Indonesia listed four rumors regarding both Jokowi and Prabowo around the time of the election: 1 Jokowi's Chinese ethnicity, 2) Jokowi's Christian parents, 3) the revival of Indonesian Communist Party (PKI), and 4) Prabowo's involvement in the abduction of activists in 1997-1998 ("Voters not swayed by online attacks on President Joko Widodo and rival Prabowo Subianto: Indonesia pollster", 2019). The first three rumors have never been able to be confirmed due to lack of evidence. The fourth rumor was sparked due to a leaked military document. However so, these rumors still have believers. What is to note is that both candidates' followers were not swayed by rumors attacking their leaders. When it came to Jokowi's rumors, Prabowo's supporters were quick to trust them. Jokowi's supporters mostly believed vice versa, anything that could bring Prabowo down.
This phenomenon is what Kahan (2012) explains as motivated reasoning. According to him, "ideologically motivated cognition" (p.407) explains how people process information influenced by their ideology and personal interests. This form of information processing makes them form and maintain beliefs, signifying their loyalty to a specific group. Therefore, disinformation, especially in a socio-political setting, thrives.

CONCLUSION
Seeing that disinformation is becoming prevalent in society, there is an urgency to face this challenge. While many people and institutions call for social media regulation, ethics for politicians, and many other measures, educators should prepare students with proper digital literacy skills. Students need to be aware of their motivated reasoning when processing new information. Apart from that, they should also know the importance of lateral reading. As the results of this research show, relying on some visible indicators to determine an online source's validity is not enough. Lateral reading is crucial in the process, and to know where to get the extra help is knowledge of its own. There are tools to verify the news, including going to the National Press Board or to Kominfo, which provides a special section to debunk viral hoaxes. Despite the convenience of these tools, time is of the essence. With more information coming in unfiltered through the Internet, it is understandably more difficult to process each feed we receive. It is then common for people to read headlines and draw conclusions that are hasty and prone to misunderstanding. To address this problem of limited time and the abundance of data, we can slow down the spread of information until we have time to process it. By following the social media campaign "think before you share", we also nurture a digitally literate Internet user's good habit.