Project: ELLIE

Researcher’s Log

1. 2026-09-06: Scrapping the journals felt slow, trying multithreading technique.
2. 2026-09-05: Journal websites in Indonesia have similar structure, but identifiers for each elements are different. Challenge for precise scrapping.
3. 2026-08-30: Links with 403 code are stashed. Proceeding with scrape procedure.
4. 2026-08-29: Some links returned 403 Forbidden. Trying HTTP Header request forgery.
5. 2026-08-28: Pseudocode for web scrapping
6. 2026-08-27: Initial draft of research report

Found Problems

403 Forbidden on Some Websites

I thought it would be appropriate for me to test whether all of the links that I have gathered are accessible. This is something that I learned from Cybersecurity courses. Manually visiting them were no problem. I could enter them just right. Things got tricky when I tried to access the journal websites using a tool (i.e., R using the httr2 package). Some of the links didn’t accept any tools or bots to access them. As a workaround (again, I learned this from Cybersecurity), I patched the HTTP Header request with custom headers. Below are the headers I tried along with the results.

The Fate of URLs with 403 Codes

As of the writing of this section (2026-08-30), I haven’t found any workaround to automate the retrieval of websites with bot protection. I will periodically update this section as I stumble upon new ways to bypass the 403 code. In the meantime, I think of doing a manual input into the database.

rvest-ing the Websites

I was naive to assume that all journal websites have identical HTML structure. Turned out, even though they use the same template website, they structure the pages differently. Some appended the URL for each issue right in the HTML text content, like the following:

...
  <a href="#">Vol XX, No XX</a>
...

...
  <div class="ex" href="#">April</div>
  <p>Vol. XX, No. XX</p>
...

I had to use wildcard instead and leave the data cleaning at a later stage.