9 posts tagged with "python"

Pydot is a diagram as code generator i.e. you write lines of code and thanks to a magic process, you can render the code as an image.

For guys just like me who are terrible at visuals, this is gold.

Let's say you have to describe an IT process such as ETL (data is loaded, one or other transformation rule is applied and the result is loaded into a database, for example), you can easily imagine to draw some rectangles using a tool like https://app.diagrams.net/ (formerly known as draw.io) or use a smarter way to do it.

Let's see how in this article.

You've two or more files and you wish to merge them. For instance, a list of employees and their salary for 2020 up to 2024. Or a list of cost centers and their budget/expenses. Or a list of students and their evaluation score. Or ...

Let's look at an example: a list of employees and their salaries.

We've many files, one by year, with this structure:

id;first_name;last_name;salary
0;Gina;Andrade;6040
1;Adam;Rowe;3866
2;Michael;Martinez;6913

And our need is to merge yearly files and generate a merged one with employees and their salary across years so, at the end, we wish to obtain this:

id;first_name;last_name;salary_2020;salary_2021;salary_2022;salary_2023;salary_2024
0;Gina;Andrade;6040;4413;7773;6208;7363
1;Adam;Rowe;3866;3678;2726;5425;7570
2;Michael;Martinez;6913;6493;5664;4902;4657

Using Pandas will make this piece of cake...

Nothing to install, nothing to configure, nothing to create first.

Here is how to use VSCode and Docker to create a ready-to-user Python environment, whatever if you're under Windows, Linux or Mac.

The only prerequisites are: we should have Docker installed on your system, you should have Visual Studio Code and you've installed the Docker for Visual Studio Code addon in VSCode.

In a previous article, I've provided some files to be able to quickly create a Python environment under Linux. Today, let's play and use the exact same files but this time under Windows.

The today challenge is easy: create a Python environment on my Windows machine without to have install Python of course and without to have to configure VSCode. Just run some magic and, voilà, as a Python newcomer, I can start to code without first losing time to configure my computer.

Over the last few weeks, I've been working on a Python script that generates PDFs. My script had to generate 70,000 of them and that obviously takes a while.

My idea was to have my script run in a Linux console and, in a second console, with a counter that increases with the number of files that have been created on the hard disk.

The first script will be in Python and I wanted something ultra-simple and using a simple Linux command.

The command ls folder_name | wc -l works but didn't stay running. Let's see how we can do better with inotifywait.

I recently worked on an ETL project in Python. Among other things, the script had to process JSON files that users dropped into a specific folder.

As this was a sensitive application, it was important to validate the script by submitting fake JSON files, but also to ensure the quality of the files received.

For the fake files, I used a tool like the Faker library for Python.

As you know, VSCode is a superb editor that lets you program in probably any programming language. An editor, not an IDE, because VSCode is basically a Notepad in its ultimate version.

If you want to program in Python, you'll need to install a few extensions in VSCode to be really comfortable, i.e. syntax highlighting, code navigation, code refactoring (like renaming a variable or a class), etc.

There are ‘ready-to-use’ editors like PyCharm but 1. they cost money and 2. they are specific (you won't be able to program in PHP with PyCharm; or even work easily with HTML/CSS files).

In this new article, we'll look at how to get a VSCode environment ready to use straight away for coding in Python, and as it's VSCode it's 1. free, 2. multi-purpose and 3. insanely powerful.

But what is Docker anyway? In the simplest possible terms, how can we explain what Docker is used for?

Let's take the case of a cooking recipe as a starting point.

Remember a few years ago when you wanted to learn a new programming language such as Python? What did you need before you could even start programming your first Hello World? You had to install the language on your computer; maybe you also had to get a lot of libraries/dependencies; you had to spend time configuring your machine before you could even start writing your first script.

That's all over with the advent of Docker and the concept of containers.

This time, we'll play with Python for this article. I'm discovering Python at the same time as writing this article: at this precise moment, I've never written .py scripts.