During Project D - Webshop at the Rotterdam University of Applied Sciences we were tasked to make a (fictional) webshop from scratch. We used .Net Core, React, Postgres and Elasticsearch as tech stack for this project. In this post I write about what I did during the project and what I have learned from it.
Gathering and saving product data
We were allowed to choose the products that the webshop was going to sell ourselves. The product that we chose was beer. The first task was to gather mock product data to populate a database.
I decided that a webscraper for an existing beer webshop would be the perfect solution for getting real product data. Therefore I wrote a webscraper in Python for the Dutch beer retailer Beerwulf. You can find the code for it on Github.
The webscraper uses the requests package to get the html source and then utilizes lxml to parse it. This posed some challenges, because not all of the pages had a consistent form. The consequence was that I had to write a lot of manual checks.
Later, I discovered that the website implemented an pagination API. I greatly optimized the scraper with asyncio, aiohtpp and by making use of this pagination api. The scraped data was turned into dataclasses, which were subsequently dumped to a json.
To migrate this data to a database I wrote another Python script that reads this json and writes it to an SQL database. In the end we had about a thousand products, which was quite enough.
We were responsible for the full stack engineering for the webshop. As mentioned above we used React as frontend framework. We decided rather quickly that we wanted to use SemanticUI React to swiftly build an amazing looking user interface.
The front-end uses React Router to determine which component to render. The component requests data from the backend and then displays it in some fancy way. Most of the development of the front-end was pretty straightforward. Most of the challenges were in connecting the front-end with the backend.
I did spend some time on creating a good user experience. The flow of the user has to be correct. It has to be easy to order something. Therefore I spend time on creating forms that are easy to complete. Another example of good UX is that our pages have infinite scrolling, this user does not have to click next page e.g.
Fiddling with forms
The most challenging aspect of the front-end was definitely the forms. We needed to create easy to use, good looking and secure forms. Think of user registration, guest ordering and changing personal data.
I defined verification rules for certain types of form fields. These fields were checked against these rules using regular expressions. However, things got a magnitude more complex when I introduced address completion, based on zip code and house number.
In the former situation we verified fields on the front-end and send a request to the server when the user had completed the form. Then on the server the data was verified again and further handled.
But in the new situation with address completion, requests were send even when the user was not done with the form. I implemented the feature using a service called Postcode Api which is basically a database containing all addresses in the Netherlands.
The form was now state based and a lot of edge cases were added. For example what happens when a user enters a zip code and house number that is correct but then changes it to a non existent zip? Every field focus and change event had to be tracked and a lot of state had to be managed to make sure no wrong data ended up in the database.
Users also have to be able to create an account for a webshop. I was tasked with creating a login/registration system. Luckily .Net Core contains a nifty package called Identity.
This package did a great deal of the work for me. Think of things like password hashing, database saving but also claim based user management. The first thing I did was create models for the user. These models extended the base user model of the Identity Framework.
The next challenge was creating a way that the users browser knows that a user is logged in. The straightforward way is using cookies with some session id or other variable used to identify the user.
However I wanted to try something new. Thus I decided to use JWT as a solution to this problem. A token generator was build into the backend. Now when the user was logged in, he/she would be identified using a freshly generated jwt. With React Redux this data was persisted in the browsers local storage.
One of the requirements for the webshop was that a user should be able to search for certain products. I was also tasked with implementing this feature. I chose to use ElasticSearch when I started to implement this function.
Integrating ElasticSearch is easy when you use a package such as NEST. You just map your models to documents in one or more indexes in ElasticSearch. You can run one or more clusters and use the API the Elastic Search services provides.
In this case I engineered it such that the client calls the backend and the backend calls the separate ES service.
Using Elastic Search enabled me to build lightning fast search. Also it meant that I was able to include useful features like Fuzzy Searching without much of a hassle. I would recommend Elastic Search to everyone who wants to implement search functionality but without any tears.
I learned a lot about web development during this project. This was the first time working with React. I quite enjoyed it and would use it again. Also .Net Core is very easy and painfree to work with and would also use it again to build similar projects.