A practical, production-ready solution for running scrapers as Celery tasks

Photo by Nicolas Picard on Unsplash

Some time ago, I accepted a project to build a portal for commercial users to scrape product details by submitting links to the product listings. The portal was a site built entirely with Python, and Scrapy.

As scraping pages typically takes time, it is often best to run them in the background and keep the users notified of its progress from time to time.

If you’re reading this, chances are, you are already familiar with Scrapy and/or Celery. …


Understanding How Stripe Webhook Events Are Signed

Photo by rupixen.com on Unsplash

Subscribing to Stripe webhook events to be notified when payments are done? After implementing your code, it’s time to make sure it works perfectly by coupling with some test cases. In this article, I’ll be writing about why testing Stripe webhook handlers require special care, and how to test them.

This article is written with language-agnostic in mind. This should be applicable to any programming languages or frameworks.

Understanding Why Testing Stripe Webhook Handlers Is Different

I classify webhooks into two categories: secure ones and insecure ones.

For insecure webhook, events are dispatched by third-party providers without giving us the option to verify the authenticity of the events…


Memoisation In Python

Memoisation is a common technique applied to reduce unnecessary repeated calculations in programming. This technique is frequently brought up as one of the dynamic programming techniques, and in articles, books aiming to help you get your first developer job.

In this article, you’ll:

  • learn what memoisation is
  • take a look at how memoisation can be implemented with a homemade solution
  • learn to achieve memoisation using the one-liner @lru_cache in Python
  • learn to inspect the cache info of a memoised function
  • explore the control knobs of @lru_cache
  • learn about the caveats when using @lru_cache-decorated functions

If you’re already familiar with the…


Concurrency In Python:

In “ Concurrent Programming in Python is not what you think it is”, I wrote about the notorious GIL in Python and did a simple experiment on existing concurrency mechanism in Python. Today, I’ll describe another common scenario in concurrent programming, the Producer/Consumer problem, with Python.

What is the producer/consumer scenario?

Imagine you are at a fast-food chain, in which there are an order line and a serving line. The staff in the serving line prepare the meals while their coworkers in the order line take orders from customers. In this scenario, the staff in the order line are known as producers, who produce order…


In this article, I will first walk you through the distinction between concurrent programming and parallel execution, discuss about Python built-ins concurrent programming mechanisms and the pitfalls of multi-threading in Python.

Understanding Concurrent Programming vs Parallel Execution

Concurrent programming is not equivalent to parallel execution, despite the fact that these two terms are often being used interchangeably.

Illustration of concurrency without parallelism

Concurrency is a property which more than one operation can be run simultaneously but it doesn’t mean it will be. (Imagine if your processor is single-threaded. )


Read it for free at https://melvinkoh.me.

While there are plenty of Python libraries available for data manipulation, they can be overkill for simple data transformation. Python standard library comes with functools.reduce() function, one of the most used functions in functional programming, is handy for simple data transformation. If you heard of reduce() or functional programming but are unsure of what they really are and how they could help you to write better Python code, this is the article for you.

In this tutorial, you’ll:

  • Understand the basic principles of functional programming
  • Understand what the Python reduce() function is
  • Learn how…

Originally published at https://melvinkoh.me

Your app is running happily on its own without major glitches. What happens when business requirements changed and you are now required to update your database schema?

In this article, I’ll be discussing several caveats that demand our attention when making schema changes in an app which is already been up and serving users. I’d also share some techniques I’d used to eliminate those risks.

The concepts and ideas of making backward-compatible schema changes should be universal regardless of programming languages and frameworks, I’ll be using Python Django to illustrate my points.

I’ll be starting off…


Originally published at https://melvinkoh.me.

While there are plenty of Python libraries available for data manipulation, they can be overkill for simple data transformation. Python standard library comes with functools.reduce() function, one of the most used functions in functional programming, is handy for simple data transformation. If you heard of reduce() or functional programming but are unsure of what they really are and how they could help you to write better Python code, this is the article for you.

In this tutorial, you’ll:

  • Understand the basic principles of functional programming
  • Understand what the Python reduce() function is
  • Learn how Python reduce…

Originally published at https://melvinkoh.me

Docker, along with Docker Compose are the most used tools under the DevOps category, according to The State of Developer Ecosystem 2019 survey by JetBrains. Chances are if you’re using Docker/Docker Compose for deployment, you’ll most likely be using them for local development as well. If you only use Docker for deployment but a virtual environment for local development, you may want to look into using Docker for development to reduce the parity between dev/prod environments, as suggested in the Twelve-Factor Methodology.

In this article, you’ll be learning how to use PDB (Python DeBugger) in a…


Originally published at https://melvinkoh.me

For the past few hours, I’ve been working on dockerizing a Nuxt.js server-side rendering app to be deployed to AWS Elastic Container Service (ECS). I hit a stone wall when environment variables are not properly injected on runtime on ECS.

In this post, I will walk us through steps to properly dockerize Nuxt.js SSR apps and share some of the lessons I learnt.

There are 2 major ways to dockerize Nuxt.js apps, and each of them has its shortcomings:

  1. Nuxt build on run: The Nuxt app is built after Docker image is built, when the container…

Melvin K.

Software Engineer | Maintainer of Python graphql-utilities | Creator of PrivateFileSaver.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store