Project Entry 4: Node.js Crawler — Project Weekly Progress Report

After deciding our project would have the trending programing language article feature, we need to figure out how to implement it.

We choose one of the most popular tech website Medium, which has considerable articles related to programing language. At first, we check their API documentation to find a way to access their article information.

Unfortunately, they didn’t provide API for article searching. So we came out the idea to use Web crawler to fetch posts from them. After making sure we are not violating any rules based on the robots.txt, I started to learn Web Crawler.

I tried to use Python web crawler called Scrapy, which is popular framework among Python developer. But I didn’t have any experience with Python before. Even I went through a Python tutorial from Youtube. I still had problem when using Scrapy.

So during last Sunday, I decided to change to use Node.js Crawler.

I write a post about how to use the Node.js Crawler. Feel free to take a look Node.js Crawler

For the design part, designers are working on the Wireframe & UIkit and logo guideline. And back-end developers start saving the data from an API to a JSON file to later populate our database.

We are making some progress!! See you guys on next week!