This is the 2nd part of exploration of pandas package. Part 1 can be found here: https://www.vallka.com/blog/leaning-pandas-data-manipulation-and-visualization-covid-19-in-scotland-statistics/

All data are taken from the official websate:

weekly-deaths-by-location-age-sex.xlsx

Taken from official the site:

https://www.nrscotland.gov.uk/statistics-and-data/statistics/statistics-by-theme/vital-events/general-publications/weekly-and-monthly-data-on-births-and-deaths/deaths-involving-coronavirus-covid-19-in-scotland/related-statistics

Let's quickly repeat data load process from the Part 1:

import numpy as np
import pandas as pd

data_2021 = pd.read_excel ('https://www.nrscotland.gov.uk/files//statistics/covid19/weekly-deaths-by-location-age-sex.xlsx',
 sheet_name='Data',
        skiprows=4,
        skipfooter=2,
        usecols='A:F',
        header=None,
        names=['week','location','sex','age','cause','deaths']
        )

data_2021['year'] = data_2021.week.str.slice(0,2).astype(int)
data_2021['week'] = data_2021.week.str.slice(3,5).astype(int)

data_1519 = pd.read_excel ('https://www.nrscotland.gov.uk/files//statistics/covid19/weekly-deaths-by-location-age-group-sex-15-19.xlsx',
        sheet_name='Data',
        skiprows=4,
        skipfooter=2,
        usecols='A:F',
        header=None,
        names=['year','week','location','sex','age','deaths'])

data_1519['cause'] = 'Pre-COVID-19'

data=data_1519.copy()
data=data_1519.append(data_2021,ignore_index=True)
data

One note here. In part 1 I was using a simple assignment od DateFrame for making a copy. This is wrong. Simple assignment does not create a copy of a DataFrame, it just creates another reference to it, so all changes to 2nd DataFrame affects the original one. To make things even more interesting, pandas provides two versions of copy() function - copy(deep=False) and copy(deep=True) deep=True is default. Here is some discussion about all three.

It is not clear for me now what is the difference between simple assignment and shallow copy() (deep=False), nor my results confirm this discussion. But deep copy() (with default parameter deep=True) seems to be working as expected all the time.

Let's see the original data_1519 to ensure we didn't modify it accidently:

data_1519

Let's quickly create totals

totals = data.groupby(['year','week']).agg({'deaths': np.sum})
totals.loc[15,'deaths_15']=totals['deaths']
totals.loc[16,'deaths_16']=totals['deaths']
totals.loc[17,'deaths_17']=totals['deaths']
totals.loc[18,'deaths_18']=totals['deaths']
totals.loc[19,'deaths_19']=totals['deaths']
totals.loc[20,'deaths_20']=totals['deaths']
totals.loc[21,'deaths_21']=totals['deaths']
totals

Let's get rid of multi-index and trnasform it into additional columns - I'm still thinking this is the quickest way of plotting data in a single plot:

totals.reset_index(inplace=True)
totals
totals.plot(x='week',y=['deaths_15','deaths_16','deaths_17','deaths_18','deaths_19','deaths_20','deaths_21'],
    title='Total deaths by week',figsize=(12,8))

Again, let's look at current year only. Ok, a couple of weeks passed since I wrote Part 1. How many weeks in this year now? And more generic question, how many weeks in each year?

wpy = totals.groupby('year').agg({'week':np.max})
wpy['week']

totals[totals['week']<=wpy['week'][21]].plot(x='week',
    y=['deaths_15','deaths_16','deaths_17','deaths_18','deaths_19','deaths_20','deaths_21'],
    title='Total deaths by week',figsize=(12,8))

Let's move on and add cumulative values (or 'running totals'). Thanks to pandas, it is very easy - comparing to SQL. (There should be other ways to achieve the same result. I used the one which seemed the most simple for me on this stage of learning)

totals['cumdeaths_15']=totals.groupby('year').agg({'deaths_15':np.cumsum})
totals['cumdeaths_16']=totals.groupby('year').agg({'deaths_16':np.cumsum})
totals['cumdeaths_17']=totals.groupby('year').agg({'deaths_17':np.cumsum})
totals['cumdeaths_18']=totals.groupby('year').agg({'deaths_18':np.cumsum})
totals['cumdeaths_19']=totals.groupby('year').agg({'deaths_19':np.cumsum})
totals['cumdeaths_20']=totals.groupby('year').agg({'deaths_20':np.cumsum})
totals['cumdeaths_21']=totals.groupby('year').agg({'deaths_21':np.cumsum})

totals
totals.plot(x='week',
   y=['cumdeaths_15','cumdeaths_16','cumdeaths_17','cumdeaths_18','cumdeaths_19','cumdeaths_20','cumdeaths_21'],
     title='Total deaths by week',figsize=(12,8))

For a year-wide values year 2020 is definitely the worst. Also on this graph we can clearly see the difference of lengths of different years (in weeks), so it is not quite correct to compare year 2020 with 53 weeks with year 2019 with only 52 weeks. Which year is worse on this graph - 2015 or 2018?

Again, let's take the beginning of year:

totals[totals['week']<=wpy['week'][21]].plot(x='week',
   y=['cumdeaths_15','cumdeaths_16','cumdeaths_17','cumdeaths_18','cumdeaths_19','cumdeaths_20','cumdeaths_21'],
   title='Total deaths by week',figsize=(12,8))

Ok, what's next? I think it would be interesting to look at year-length data back from the current date. That is, from March 2020 to March 2021 - and compare these data to the previous years. How to do this? Not obvious...

I came with a solution with building 'negative weeks' data. Let's call last week of 2020 - week 0 of 2021, week 52 of 2020 - week -1 of 2021, week 51 of 2020 - week -2 of 2021, and so on. We'll add these new 'negative' weeks to our dataframe, effectively doubling the data. We'll drop all the calculated columns now, we'll re-calculate them later:

totals1 = totals.reindex(columns=['year','week','deaths'])
totals1
totals0 = totals1.copy()
totals0.loc[(totals0.year==20),'week0'] = (totals0.loc[totals.year==20,'week']-wpy.week[20]).astype(int)
totals0.loc[(totals0.year==19),'week0'] = (totals0.loc[totals.year==19,'week']-wpy.week[19]).astype(int)
totals0.loc[(totals0.year==18),'week0'] = (totals0.loc[totals.year==18,'week']-wpy.week[18]).astype(int)
totals0.loc[(totals0.year==17),'week0'] = (totals0.loc[totals.year==17,'week']-wpy.week[17]).astype(int)
totals0.loc[(totals0.year==16),'week0'] = (totals0.loc[totals.year==16,'week']-wpy.week[16]).astype(int)
totals0.loc[(totals0.year==15),'week0'] = (totals0.loc[totals.year==15,'week']-wpy.week[15]).astype(int)
totals0.loc[(totals0.year==20),'year0'] = totals0.loc[totals.year==20,'year']+1
totals0.loc[(totals0.year==19),'year0'] = totals0.loc[totals.year==19,'year']+1
totals0.loc[(totals0.year==18),'year0'] = totals0.loc[totals.year==18,'year']+1
totals0.loc[(totals0.year==17),'year0'] = totals0.loc[totals.year==17,'year']+1
totals0.loc[(totals0.year==16),'year0'] = totals0.loc[totals.year==16,'year']+1
totals0.loc[(totals0.year==15),'year0'] = totals0.loc[totals.year==15,'year']+1

totals0 = totals0.loc[totals0['year0']>0]
totals0

A few more manipulations with columns...

neworder =['year0','week0','deaths']
totals0 = totals0.reindex(columns=neworder)
totals0.rename(columns={'year0':'year','week0':'week'},inplace=True)
totals0

Now we have totals1 with 'normal' weeks and totals0 with 'negative' weeks. Let's just combine them:

totals_x = totals0.copy()
totals_x = totals_x.append(totals1)
totals_x.reset_index(inplace=True)
totals_x

Now we'll repeat our exercise with adding columns for each year (there should be a better way of doing this, but let's not changing it for now)

totals_x.loc[totals_x['year']==15,'deaths_15']=totals_x['deaths']
totals_x.loc[totals_x['year']==16,'deaths_16']=totals_x['deaths']
totals_x.loc[totals_x['year']==17,'deaths_17']=totals_x['deaths']
totals_x.loc[totals_x['year']==18,'deaths_18']=totals_x['deaths']
totals_x.loc[totals_x['year']==19,'deaths_19']=totals_x['deaths']
totals_x.loc[totals_x['year']==20,'deaths_20']=totals_x['deaths']
totals_x.loc[totals_x['year']==21,'deaths_21']=totals_x['deaths']

totals_x.sort_values(['year','week'],inplace=True)

totals_x

We also need to re-sort the data, otherwise our line graphs will have gaps in unexpected places.

Now we can draw the graph for 2 years:

totals_x.plot(x='week',
    y=['deaths_15','deaths_16','deaths_17','deaths_18','deaths_19','deaths_20','deaths_21'],
        title='Total deaths by week',figsize=(12,8))

The lines repeat themselves. Yes, this is what we wanted.

Now we can simply take the last 53 weeks from this 2-year wide data:

max_wk = 53
totals_x.loc[(totals_x['week']<=wpy.week[21])&(totals_x['week']>wpy.week[21]-max_wk)].plot(x='week',
    y=['deaths_15','deaths_16','deaths_17','deaths_18','deaths_19','deaths_20','deaths_21'],
        title='Total deaths by week',figsize=(12,8))

Let's calculate running totals for this period. Not so straightforward, but still doable:

max_wk=53
totals_x1 = totals_x.loc[(totals_x['week']<=wpy.week[21])&(totals_x['week']>wpy.week[21]-max_wk)].copy()

totals_x1['cumdeaths_15']=totals_x1.groupby('year').agg({'deaths_15':np.cumsum})
totals_x1['cumdeaths_16']=totals_x1.groupby('year').agg({'deaths_16':np.cumsum})
totals_x1['cumdeaths_17']=totals_x1.groupby('year').agg({'deaths_17':np.cumsum})
totals_x1['cumdeaths_18']=totals_x1.groupby('year').agg({'deaths_18':np.cumsum})
totals_x1['cumdeaths_19']=totals_x1.groupby('year').agg({'deaths_19':np.cumsum})
totals_x1['cumdeaths_20']=totals_x1.groupby('year').agg({'deaths_20':np.cumsum})
totals_x1['cumdeaths_21']=totals_x1.groupby('year').agg({'deaths_21':np.cumsum})

totals_x1.plot(x='week',
        y=['cumdeaths_16','cumdeaths_17','cumdeaths_18','cumdeaths_19','cumdeaths_20','cumdeaths_21'],
                title='Total deaths by week',figsize=(12,8))

Well done pandas.

Ok, it's more or less obvious now that plotting all 7 years on the same graph is really messy. So let's just 'compress' years 2015-2019 into simple average line:

avg = totals[totals['year']<20].groupby('week').agg({'deaths':np.average,'year':np.max})

avg['year'] = 1519
avg.rename(columns={'deaths':'deaths_1519'},inplace=True)
avg.reset_index(inplace=True)
avg

We called this 'year 1519'. Let's add this new average year to the same data we have:

totals=totals.append(avg)
totals

Let's quickly plot the new line together with the original years, just to make sure we got it right:

totals[(totals['week']<=wpy['week'][21])&(totals['week']>0)].plot(x='week',
    y=['deaths_15','deaths_16','deaths_17','deaths_18','deaths_19','deaths_1519',],
        title='Total deaths by week',figsize=(12,8))

Looks right - the new line goes somewhere between other years.

Let's plot the years of interest together with the average:

totals.plot(x='week',
    y=['deaths_1519','deaths_20','deaths_21'],
        title='Total deaths by week',figsize=(12,8))

What's about 53 weeks backwards from the current date? Let's repeat the calculation we already did, and get the result:

totals_x=totals_x.append(avg)
totals_x[totals_x['year']==1519]
avg0=avg.copy()
avg0['week']=avg0['week']-53
avg0
totals_x=totals_x.append(avg0)
totals_x.sort_values(['year','week'],inplace=True)
max_wk = 53
totals_x.loc[(totals_x['week']<=wpy.week[21])&(totals_x['week']>wpy.week[21]-max_wk)].plot(x='week',
                        y=['deaths_1519','deaths_21'],title='Total deaths by week',figsize=(12,8))

We can see that we are clear below the average for at least a couple of weeks now. Ok, let's stop for now. Well done pandas.

Next time we'll try to add some interactivity to the charts.

The code for today's article can be found on gist.github and Google Colab:

https://gist.github.com/vallka/621ea43c236f8f24f2c589190d5ca07f

https://colab.research.google.com/drive/1a5FyWN5psehoqnAUEev0ODE2KNMiFTTE?usp=sharing

Let's do a little bit of pandas. Pandas is (or are? :) extremely popular. Let's just dive in.

First of all, using pandas is the easiest way to open an Excel file. Just one line of code. Let's take, for example, this file -

Weekly deaths by location of death, age group, sex and cause, 2020 and 2021
(10 March 2021)

weekly-deaths-by-location-age-sex.xlsx

Taken from official the site:

https://www.nrscotland.gov.uk/.../related-statistics

import pandas as pd

data = pd.read_excel ('https://www.nrscotland.gov.uk/files//statistics/covid19/weekly-deaths-by-location-age-sex.xlsx',
 sheet_name='Data',
        skiprows=4,
        skipfooter=2,
        usecols='A:F',
        header=None,
        names=['week','location','sex','age','cause','deaths']
        )

data

Read more...

Prestashop Modules Programming. Bcc outgoing emails

Published Feb. 28, 2021, 8:10 p.m.

What is the idea?

Let's start programming Prestashop modules. Where to start from? Here is the documentation:

https://devdocs.prestashop.com/1.7/modules/

Ok. But what would be our first module? It should be very simple, and yet it should be useful. Here is the idea: add BCC to all outgoing emails, that we will have copies of all outgoing emails in out mailbox. Why would we want it? Actually this is what many people want, just try to google. For newly set up website it may be useful. You may want to see how actually the emails look like. And find all versions of emails which Prestashop sends to your customers. Probably after looking to these emails you would want to change them a bit, maybe remove a default "Powered by Prestashop" line at the bottom... You may want to keep monitoring outgoing emails to ensure that next update of Prestashop do not revert your custom emails to default ones (you probably forgot to check "Keep email templates" while updating Prestashop). Ok, let's do it.

Among Prestashop hooks there is one which can be used for our purpose: actionEmailSendBefore

actionEmailSendBefore

Before sending an email This hook is used to filter the content 
or the metadata of an email before sending it or even prevent its sending

Located in: /classes/Mail.php

https://devdocs.prestashop.com/1.7/modules/concepts/hooks/list-of-hooks/

If we have a chance to look inside /classes/Mail.php file, we will see that all the parameters are prefixed with "&", what means we can change them:

$hookBeforeEmailResult = Hook::exec(
        'actionEmailSendBefore',
        [
            'idLang' => &$idLang,
            'template' => &$template,
            'subject' => &$subject,
            'templateVars' => &$templateVars,
            'to' => &$to,
            'toName' => &$toName,
            'from' => &$from,
            'fromName' => &$fromName,
            'fileAttachment' => &$fileAttachment,
            'mode_smtp' => &$mode_smtp,
            'templatePath' => &$templatePath,
            'die' => &$die,
            'idShop' => &$idShop,
            'bcc' => &$bcc,
            'replyTo' => &$replyTo,
        ],
        null,
        true
    );

I couldn't find any place in Prestashop Back Office which could potentially use BCC. So we can assume (for simplicity) it is always empty and we can just set it to our own value in the hook.

Let's create a module!

Luckily, there is a simple way to create a module skeleton. There is a module generator supplied by Prestashop itself:

https://validator.prestashop.com/generator

I wouldn't say it works perfectly but it works. Probably, it is not updated as ofter as new Prestashop versions are released, and contains some bugs. Also, as mentioned in the documentation, if you want to create a Payments module for Prestashop 1.7 you should not use this generator. But for our purpose it as just invaluable.

So put some values to start a module - name, version, author name etc.:

Read more...

How to deploy Django application to HostPresto!

Published Feb. 12, 2021, 6:26 p.m.

In this tutorial we will deploy Django Polls project from official Django tutorial to the HostPresto!

If you have started learning Django framework, you should be familiar with this tutorial project. Moreover, you should have this project on your local machine up and running, using Django built-in development webserver. Now we will deploy this project to HostPresto!

If you do not have the project on your machine, you may just clone my GitHub repository:

https://github.com/vallka/django-polls/

(How to run the Polls project locally is out of scope of this tutorial. Please refer to official docs to set up local development environment and build the project: https://docs.djangoproject.com/en/3.0/intro/tutorial01/. )

Here we will assume that you have the project set up into your local virtual environment. The project should have the following or similar structure on your machine:

└───mysite
 │   db.sqlite3
 │   manage.py
 │   README.md
 │   requirements.txt
 ├───mysite
 │       settings-production.py.sample
 │       settings.py
 │       urls.py
 │       wsgi.py
 │       __init__.py
 ├───polls
 │   │   admin.py
 │   │   apps.py
 │   │   models.py
 │   │   tests.py
 │   │   urls.py
 │   │   views.py
 │   │   __init__.py
 │   ├───migrations
 │   │       0001_initial.py
 │   │       __init__.py
 │   ├───static
 │   │   └───polls
 │   │       │   style.css
 │   │       └───images
 │   │               background.jpg
 │   └───templates
 │       └───polls
 │               detail.html
 │               index.html
 │               results.html
 └───templates
     └───admin
             base_site.html

Let’s start to set up our HostPresto! Environment.

Creating subdomain

Suppose you have a domain registered with HostPresto! - yourname.xyz (BTW .xyz domains are the cheapest :). For this tutorial, we will create a subdomain polls.youname.xyz We think this is the most natural and straightforward method. Setting up the tutorial project in the root domain may pollute your space. You may also have some website sitting there already. Adding a subdirectory like www.youname.xyz/polls is less convenient, in our opinion.

Log in into cPanel and click on Subdomains in DOMAINS section. Enter ‘polls’ into Subdomain field. Select your main domain (youname.xyz) from Domain dropdown. Document Root will be auto populated with ‘polls.yourname.xyz’. Let’s accept the default value. Click Create.

subdomains

Subdomain created. If you point your browser to http://polls.yourname.xyz, most probably you will see a directory listing – something like this:

Index of

This is how an empty website looks like on HostPresto!

Setting up Python application

In cPanel go to SOFTWARE section and select Setup Python App. Click on CREATE APPLICATION.

HostPresto! Uses software from Passenger (https://www.phusionpassenger.com/) to set up Python applications. The setup page isn’t very intuitive, in my opinion. But let’s use it anyway.

Select the latest Python version in the first field. It is 3.7.3 at the moment of writing. In Application root type in the directory name we have created in the previous step, relatin to your home directory – that is, just ‘polls.yourname.xyz’. Select ‘polls.yourname.xyz’ as Application URL in the next line. Leave Application startup file and Application Entry point empty – the system will create this file by its own rules and let it do it. Last thing – it is sensible to name a log file (Passenger log file) – enter /home/yourname/polls.log

Create application

Click on CREATE

Click in START APP

Again, point your browser to http://polls.yourname.xyz (or refresh page if you still have it in one of the tabs). Now it should show something like this:

It works

This means we have successfully created a virtual environment for our Python application.

Uploading project files.

To upload files we will use any FTP client. You can find all needed information (FTP host, port, credentials) in FILES – FTP Accounts section of cPanel. Remember to use cPanel username and password (the ones you can find on your Welcome email), not HostPresto! Client area email/password. (You can also connect via SFTP using the same sPanel credentials. Please notice that you need to use port 88 for SFTP, not standard port 22. ) You can even use cPanel File Manage to upload files, but this is too cumbersome, in my opinion.

Navigate to newly created directory polls.yourname.xyz. This will be the root directory for our project, corresponding to ‘outer mysite/’ directory mentioned in Django Polls tutorial. Upload the whole structure here. (Ensure that manage.py file resides directly in polls.yourname.xyz directory)

We will need requirements.txt file which should also live in root project directory - polls.yourname.xyz. Most probably you already have this file in you project directory on your local machine, so it is already uploaded to the server. But if there is no such file, you need to create it. For this, go to your project directory on your local machine, activate your project virtual environment (running command source /path/to/ENV/bin/activate or similar method), then run command:

pip freeze >  requirements.txt

This will create requirements.txt file. The content if this file should look something like this:

asgiref==3.2.3
Django==3.0
pytz==2019.3
sqlparse==0.3.0

Numbers may differ, there may be more lines. It shouldn’t matter for now.

Upload this file to root project directory (polls.yourname.xyz) if it is not already there.

Installing Python packages

Once we have a virtual environment, we have to install needed packages. How to do this, is not particularly clear from cPanel web interface. Yes, we can do it directly in web interface (we need to type ‘requirements.txt’ - if we have this file in root subdirectory – in ‘Configuration files’ section, click ADD, then click ‘Run Pip install'), but we won’t do it this way. Let’s do it in traditional way, running shell command. We will need to use shell commands anyway, so let’s go to command line straight away.

We need to make a ssh connection to our server. If you are on Windows, I recommend to use PuTTY program. Use your cPanel username and password (not client area password!) and change port 22 to 88 – HostPresto uses port 88 for ssh connection.

Once logged in, activate your project virtual environment and cd into project directory. How to do this? Actually cPanel contains a useful hint: at the top of the page the command is displayed:

Enter to the virtual environment. To enter to virtual environment, run the command:

source /home/yournamexy/virtualenv/polls.yourname.xyz/3.7/bin/activate && cd /home/yournamexy/polls.yourname.xyz

Just click on command and it will be copied to the clipboard.

Paste the command to PuTTY terminal window and run it – now you are in your project directory and virtual environment is activated. First of all, let’s upgrade pip – run the command

pip install --upgrade pip

Now we can install python packages needed for the project. Remember, we have a list in requirements.txt file. Run the following command:

pip install -r requirements.txt

After some busy work of the system, all needed packages are installed.

First attempt to run

You should already know that first we need to (re)create a database. For this, as you know, the following command needed:

python manage.py migrate

Run it. Most probably, you will get a long list of errors. The very last line contains the actual error, something like this:

django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).

Actually, this is good that we run into this error in the very first step. (If not and your migration completed successfully you still need to read further!). As you may remember, for local development you have used SQlite server, built-in Python SQL server. This is completely fine for local development, but totally unacceptable for production server (and we are pretending to deploy to production server now). So luckily for us, Django doesn’t like the version of installed SQlite server, so there is no other way than to switch to MySQL. (Traditionally Django uses PostgreSQL but in our case we have MySQL pre-installed, so we will use MySQL)

So let’s return to cPanel and create a MySQL databse, using available tools.

Creating MySQL database for the project

This is easy. In cPanel, go to DATABASES – MySQL Database Wizard. Type in a name for a database. Let’s call it 'polls' in order to be consistent. cPanel automatically prefixes the database name with yournamexy_, so the full name will be 'yournamexy_polls'. Click on Next Step, provide a name and a password for new database user (let’s give the user the same name: 'yournamexy_polls'). Provide a password. Click Next. On Step 3 check ALL PRIVILEGES. Click Next.

Don’t forget newly created password – we’ll need it in a minute.

Creating settings-production.py file

Now we have to alter our settings.py file – change SQlite database with MySQL. How will we do it?

Updating the settings.py file itself is not a good idea. Yes, we can do it, directly on the server, but we won’t be able to use this file on a local computer. How to solve it? Usually a project has two or more settings.py files – one for dev server, one for production, possibly one for staging server. We will create a second file for production - let’s call it settings-production.py

I recommend to do it on your local machine and upload it on the server afterwards. So navigate ‘inner mysite/’ directory on you local machine. File settings.py should be there. Alongside this file create a new file settings-production.py. Put the following line at the top of the file:

from . settings import *

We simply import all settings from our existing settings.py file. Now we need to override some of them. Let’s do it one by one, to make things clear, what and why we are overriding.

First of all, we need to change DATABASES. So add the following lines:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'yournamexy_polls',
        'USER': 'yournamexy_polls',
        'PASSWORD': os.environ['POLLS_DB_PASSWORD'],
        'HOST': 'localhost',
    }
}

(Replace yournamexy_polls with your own name yor have created in previous step)

It’s not a good idea to hardcode the database password, so we will read it from environment variable.

Upload the file to the server

Running manage.py on production server

How should we tell Django to use setting-production.py instead of settings.py? All we need is to set an environment variable DJANGO_SETTINGS_MODULE

So run in ssh session:

export DJANGO_SETTINGS_MODULE=mysite.settings-production

Remember, we also need to set a variable for a password:

export POLLS_DB_PASSWORD=<your password>

(Use a password you have chosen when creating a database)

Now we are ready to run migration in production environment. Run agan:

python manage.py migrate

Now it should finish successfully.

(You may found out that it ended with error - complaining that 'package mysqlclient not found'. In this case you simple need to install needed package:

pip install mysqlclient

)

Create a superuser:

python manage.py createsuperuser

Now we need to finish configuring the Python applications

Configuring passenger_wsgi.py

Return to cPanel – Setting Python App page.
You will see that Application startup file and Application Entry point fields are populated now.
Application startup file is called passenger_wsgi.py. Find this file in the root folder of our project. Open it in File Manager. It should look like this:

import os
import sys

sys.path.insert(0, os.path.dirname(__file__))

def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    message = 'It works!\n'
    version = 'Python %s\n' % sys.version.split()[0]
   response = '\n'.join([message, version])
   return [response.encode()]

We don’t need all this. So delete the content and replace it with the following:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings-production')
application = get_wsgi_application()

(if you have cloned my GitHub project, you should have a file called passenger_wsgi.py.sample. It is called .sample on purpose – the system will auto-create this file for us, so initial content would be overwriten anywat. So rename the files or copy-paste the content from passenger_wsgi.py.sample to passenger_wsgi.py)

As you can see, DJANGO_SETTINGS_MODULE variable is set here. We also need to set out db password. We can do it here as well (what is not a very good idea if we going to check in this file to GitHub), or set up POLLS_DB_PASSWORD variable in cPanel. What we’ll do in Environment variables section of Python Application page. Click ADD VARIABLE, type in POLLS_DB_PASSWORD and your password, click done. Then click SAVE on top of the page. Finally, click RESTART

Point your browser to http://polls.yourname.xyz and refresh page. Most probably you will see:

DisallowedHost at /

Invalid HTTP_HOST header: 'polls.yourname.xyz'. You may need to add 'polls.yourname.xyz' to ALLOWED_HOSTS.

Thankfully, the message is very clear. ALLOWED_HOSTS = [] in settings.py, what is fine for development. Now we need to override this settings for production. Open settings-production.py file and add the line:

ALLOWED_HOSTS = ['polls.yourname.xyz']

(Replace 'yourname.xyz' with your actual subdomain)

RESTART application again

Page not found (404)
Request Method:     GET
Request URL:    http://polls.yourname.xyz/

Ok, this is expected. Remember, this is no ‘/’ path defined in urlpatterns in urls.py (unless you have added it of course). Navigate to http://polls.yourname.xyz/polls -

No polls are available.

This is exactly what we expected to see. Navigate to

http://polls.vallka.xyz/admin

You can log in using superuser username and password and create some polls now.

Static Files

Ok, everything seems to be working now... but where is my background? And the admin panel looks like it lost all css styles... In fact it did.

Remember, .css, .js, image files are not served by default by Django on production server. They are on local development server, but to use them on production a few more steps needed. Actually, there are two ways to add static files to production website – an easy one and... not so easy one, but recommended by Django and supported out of box.

Let’s start with the second one, more complicated default Django way. This will give us more understanding of Django.

According to Django philosophy, all static files must be placed to another webserver. Let’s simulate this another webserver with a second subdomain. Let’s create a subdomain called ‘polls-static’. On cPanel, go to cPanel and click on Subdomains in DOMAINS section. Enter ‘polls-static’ into Subdomain field. Select your main domain (youname.xyz) from Domain dropdown. Accept the auto populated value for subdomain root - ‘polls-static.yourname.xyz’. Click Create.

Now we have a separate subdomain where we will put out static files. Open settings-production.py in File Manager – Edit file and add two lines:

STATIC_ROOT = '/home/yournamexy/polls-static.yourname.xyz/'
STATIC_URL = 'http://polls-static.yourname.xyz/'

You may remember that settings.py already contains a line:

STATIC_URL = '/static/'

In settings-production.py we have overridden this value with a new one. STATIC_ROOT is a new setting, we didn’t have it at development, as it wasn't needed.

Now go to SSH terminal. If you logged of, log back in. Important! Do not forget activate and cd into you virtual environment and export environment variables:

source /home/younamexy/virtualenv/polls.youname.xyz/3.7/bin/activate && cd /home/vallkaxy/polls.yourname.xyz
export DJANGO_SETTINGS_MODULE=mysite.settings-production

export POLLS_DB_PASSWORD=<your password>

These commands are saved in bash history, so it will be easy to repeat them. (You may add them to your .bashrc if you want them to be executed automatically upon log on)

Now run the following command:

python manage.py collectstatic

Type 'yes' if prompted

After a while you will see:

132 static files copied to '/home/yournamexy/polls-static.yourname.xyz'

All static files from different subdirectories of polls project (including admin files) are now copied to '/home/yournamexy/polls-static.yourname.xyz/' directory. You can open it in File Manager and see it.

Now restart application on Python Application page

Refresh polls.yourname.xyz/polls and polls.yourname.xyz/admin

Now all should be displayed exactly as it was on your local development server.

Look ar page source of /polls/ page. You may find a line:

<link rel="stylesheet" type="text/css" href="http://polls-static.yourname.xyz/polls/style.css">

If you have worked on local development server, you may recall that the same line looked like this:

<link rel="stylesheet" type="text/css" href="/static/polls/style.css">

You see, now, on production, style.css file (and other static files) is server from different subdomain.

This is how it supposed to work with Django by default.

However, there is a simpler way.

Using WhiteNoise package for serving static files

All this additional setup with additional polls-static subdomain was required because Django wants us to put Python code and static files on separate webservers – in the sake of efficiency. However, in our case, these two subdomains are still served by the same webserver. So why all these additional efforts?

Luckily, adding one more package - WhiteNoise – allows to reduce a headache. Go to ssh terminal and run:

pip install whitenoise

Next, edit settings.py – not settings-production.py, the main settings.py file. WhiteNoise works equally well in development and production environments. Add a line to MIDDLEWARE:

MIDDLEWARE = [
 'django.middleware.security.SecurityMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',  # new
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 ]

(it should be placed directly after 'django.middleware.security.SecurityMiddleware' according to WhiteNoise documentation)

Add this line at the end of the file, after STATIC_URL:

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

‘staticfiles’ is a special name, known to WhiteNoise

Now comment out (or simply delete) the two lines in settings-production.py we added previously:

#STATIC_ROOT = '/home/yourname/polls-static.yourname.xyz/'
#STATIC_URL = 'http://polls-static.yourname.xyz/'

We don’t need them any more.

In ssh terminal, run collectstatic again:

python manage.py collectstatic

Restart application and refresh application pages. Now you will see the same working application, and references to static files in page source will look exactly as on development server, e.g.:

<link rel="stylesheet" type="text/css" href="/static/polls/style.css">

But don’t think that WhiteNoise package is for simplifying stuff only. If you really need it, with just a few configuration variables you can set up your static files on different server, as before, or on cloud storage or CDN. So it can be benefitial for performance also if set up accordingly.

One last step

On production server, we shouldn't forget to switch DEBUG off. So add the final line to settings-production.py:

DEBUG = False

All done.

We won’t go further in this tutorial. Now we have successfully deployed our application on HostPresto! Server, and learned two different methods of serving static files.


HostPresto! is reasonably priced and reliable hosting. What makes it outstanding — like many hosting providers they give you PHP and MySQL out of the box, but unlike most of other providers, they give you a very simple solution to host almost any kind of application, Node.js, Django, Flask, Ruby-on-Rails to mention a few. And all this zoo of applications can co-exist on the same server! What makes HostPresto! a great choice for any developer.


HostPresto are one of the leading independent, affordable website hosts.
hostpresto.com

In this tutorial we will create a brand new Django application on HostPresto!

Let’s start to set up our HostPresto! Environment.

Creating subdomain

Suppose you have a domain registered with HostPresto! - yourname.xyz (BTW .xyz domains are the cheapest :). For this tutorial, we will create a subdomain start.youname.xyz We think this is the most natural and straightforward method. Setting up the tutorial project in the root domain may pollute your space. You may also have some website sitting there already. Adding a subdirectory like www.youname.xyz/start is less convenient, in our opinion.

Log in into cPanel and click on Subdomains in DOMAINS section. Enter ‘start’ into Subdomain field. Select your main domain (youname.xyz) from Domain dropdown. Document Root will be auto-populated with ‘start.yourname.xyz’. Let’s accept the default value. Click Create.

subdomains

Subdomain created. If you point your browser to http://start.yourname.xyz, most probably you will see a directory listing – something like this:

Index of

This is how an empty website looks like on HostPresto!

Setting up a Python application

In cPanel go to the SOFTWARE section and select Setup Python App. Click on CREATE APPLICATION.

HostPresto! Uses software from Passenger (https://www.phusionpassenger.com/) to set up Python applications. The setup page isn’t very intuitive, in my opinion. But let’s use it anyway.

Select the latest Python version in the first field. It is 3.7.3 at the moment of writing. In Application root type in the directory name we have created in the previous step, relating to your home directory – that is, just ‘start.yourname.xyz’. Select ‘start.yourname.xyz’ as Application URL in the next line. Leave Application startup file and Application Entry point empty – the system will create this file by its own rules and let it do it. The last thing – it is sensible to name a log file (Passenger log file) – enter /home/yourname/start.log

Create application

Click on CREATE

Click in START APP

Again, point your browser to http://start.yourname.xyz (or refresh the page if you still have it in one of the tabs). Now it should show something like this:

It works

This means we have successfully created a virtual environment for our Python application.

Installing Python packages

Once we have a virtual environment, we have to install needed packages. How to do this, is not particularly clear from cPanel web interface. Yes, we can do it directly in web interface (we need to type ‘requirements.txt’ - if we have this file in root subdirectory – in ‘Configuration files’ section, click ADD, then click ‘Run Pip install'), but we won’t do it this way. Let’s do it in traditional way, running shell command. We will need to use shell commands anyway, so let’s go to command line straight away.

We need to make an ssh connection to our server. If you are on Windows, I recommend using PuTTY program. Use your cPanel username and password (not client area password!) and change port 22 to 88 – HostPresto uses port 88 for ssh connection.

Once logged in, activate your project virtual environment and cd into the project directory. How to do this? Actually cPanel contains a useful hint: at the top of the page the command is displayed:

Enter to the virtual environment. To enter to virtual environment, run the command: source /home/yournamexy/virtualenv/start.yourname.xyz/3.7/bin/activate && cd /home/yournamexy/start.yourname.xyz

Just click on command and it will be copied to the clipboard.

Paste the command to PuTTY terminal window and run it – now you are in your project directory and virtual environment is activated. First of all, let’s upgrade pip – run the command

pip install --upgrade pip

Next, we will install Django:

pip install django

We might need mysqlclient, so let's install it:

pip install mysqlclient

I highly recommend to install WhiteNoise for managing static files, this will make life much easier:

pip install whitenoise

To finalize set up, let's create requirements.txt:

pip freeze > requirements.txt

This command will create requirements.txt file, which can be used to re-create the environment if needed. We don't need this file right now right here, but it's a good practice to have it at hand.

Creating a project

Run the following command:

django-admin startproject start .

(Please notice the dot at the end of command. This dot tells django-admin to create a project in the current directory. Otherwise, it will create a subdirectory 'start' in the current directory - this is not what we want.)

Now let's try to create an app:

python manage.py startapp startapp

Most probably, you will get a long list of errors. The very last line contains the actual error, something like this:

django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later is required (found 3.7.17).

Actually, this is good that we run into this error in the very first step. (If not and your command completed successfully you still need to read further!). As you may remember, for local development you may SQlite server, built-in Python SQL server. This is completely fine for local development, but totally unacceptable for a production server (and we are pretending to create a production server now). So luckily for us, Django doesn’t like the version of the installed SQlite server, so there is no other way than to switch to MySQL. (Traditionally Django uses PostgreSQL but in our case we have MySQL pre-installed, so we will use MySQL)

So let’s return to cPanel and create a MySQL database, using available tools.

Creating MySQL database for the project

This is easy. In cPanel, go to DATABASES – MySQL Database Wizard. Type in a name for a database. Let’s call it 'start' in order to be consistent. cPanel automatically prefixes the database name with yournamexy_, so the full name will be 'yournamexy_start'. Click on Next Step, provide a name and a password for a new database user (let’s give the user the same name: 'yournamexy_start'). Provide a password. Click Next. On Step 3 check ALL PRIVILEGES. Click Next.

Don’t forget the newly created password – we’ll need it in a minute.

Configuring Django to use MySQL

In file manager, navigate to /home/yournamexy/start.yourname.xyz/start. This subdirectory should contain settings.py file. Open this file in editor and find DATABASES section. Replace the whole section with the following lines:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'yournamexy_start',
        'USER': 'yournamexy_start',
        'PASSWORD': os.environ['START_DB_PASSWORD'],
        'HOST': 'localhost',
    }
}

(Replace yournamexy_start with your own name yor have created in the previous step)

It’s not a good idea to hardcode the database password, so we will read it from environment variable.

So, return to the console and set a variable for a password:

export START_DB_PASSWORD=[your password]

(Use a password you have chosen when creating a database)

Now you should be able to create an application. Run again:

python manage.py startapp startapp

(Now I realized that I have chosen a bad name for my app - 'startapp'. startapp is also a command in manage.py, so it is unclear what is what in the above command. Much better would be to name the app, say, 'first_app', and the command to create such app will be: 'python manage.py startapp first_app')

Configuring passenger_wsgi.py

Return to cPanel – Setting Python App page.
You will see that Application startup file and Application Entry point fields are populated now.
Application startup file is called passenger_wsgi.py. Find this file in the root folder of our project. Open it in File Manager. It should look like this:

import os
import sys

sys.path.insert(0, os.path.dirname(__file__))

def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    message = 'It works!\n'
    version = 'Python %s\n' % sys.version.split()[0]
    response = '\n'.join([message, version])
    return [response.encode()]

We don’t need all this. So delete the content and replace it with the following:

import os
from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'start.settings')
application = get_wsgi_application()

Last thing we need to do before attempting to run our new Django site is to set environment variable for db password in Python web application. We did it in a command shell, but we need to do the same for the webserver.

For this, return to SOFTWARE section in cPanel, go to Setup Python Applicaiton, click on Edit (pencil) icon in a line with our newly created application start.yourname.xyz. In Environment variables section add a variable named START_DB_PASSWORD with your db password. Click SAVE at the top of the page. Click RESTART.

Now we are ready to try. Point your browser to http://start.yourname.xyz and refresh the page. Most probably you will see:

DisallowedHost at /

Invalid HTTP_HOST header: 'start.yourname.xyz'. You may need to add 'start.yourname.xyz' to ALLOWED_HOSTS.

Thankfully, the message is very clear. ALLOWED_HOSTS = [] in settings.py. This default value only allows us to run a web app on the localhost. Now we need to change this. Open settings.py file and find the line:

ALLOWED_HOSTS = []

Reolace it with the following:

ALLOWED_HOSTS = ['polls.yourname.xyz']

(Replace 'yourname.xyz' with your actual subdomain)

RESTART application again

Refresh http://start.yourname.xyz page:

success

Congratulations! Our first Django application is up and running!

Further steps

Ok, let's not stop on this. Let's go forward. Let's try admin page: http://start.yourname.xyz

admin-not-good

It looks like our web application doesn't see .css files. Yes, this is the problem. Luckily, we already installed WhiteNoise package, which will help us with just one line in settings.py. Open settings.py in editor and add one line to MIDDLEWARE section:

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',            #new
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

RESTART the app. Refresh the page:

admingood

Much better!

This is what WhiteNoise does for us - no need to run collectstatic command. All static files are being served directly from Python server.

Next thing we need to migrate the database. We didn't write a single line of code yet, but Django did a lot for us. So let's migrate:

python manage.py migrate

Let's create a superuser:

python manage.py createsuperuser

Now you can log in to admin page.

Creating some meaningfull app

Ok, let's create something meaningfull. Let it be a simple Contact list.

First of all, we need to create a model. Let's open a nearly empty file startapp/models.py and add some content. A very simple Contact model:

startapp/models.py

from django.db import models

# Create your models here.
class Contact(models.Model):
    forename = models.CharField(max_length=100)
    surname = models.CharField(max_length=100)
    email = models.EmailField(max_length=100)

    def __str__(self):
        return f"{self.forename} {self.surname}: {self.email}"

We need to update settings.py one more time. We need to add our app, startapp, to INSTALLED_APP

start/settings.py

...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'startapp',   #new
]
...

Let's add our new model to admin part. Add these lines to startapp/admin.py

startapp/admin.py

from django.contrib import admin

# Register your models here.
from .models import Contact
admin.site.register(Contact)

In the console, run following commands:

python manage.py makemigrations
python manage.py migrate

RESTART the app and login to admin interface, using credentials you supplied to createsuperuser command

admin

In this tutorial, I won't go further into creating views, forms and other common stuff. All this is common to any hosting provider, so you may find tons of useful tutorials on the Internet. Instead, I'd like to pay some attention to details, specific for HostPresto. Namely, testing.

Testing our app

Let's create a very simple test in startapp/tests.py

startapp/tests.py

from django.test import TestCase

# Create your tests here.
from .models import Contact

class ContactModelTests(TestCase):    
    def test_contact_save(self):
        contact = Contact()
        contact.save()

        self.assertNotEqual(contact.id, None,'id is autocreated')
        self.assertEqual(len(Contact.objects.all()),1,'Exactly 1 record in db after save')

If we try to run test the usual way:

python manage.py test

Most likely, we will get an error, something like this:

Got an error creating the test database: 
(1044, "Access denied for user 'yournamexy_start'@'localhost' to database 'test_yournamexy_start'")

This is because Django tries to create a new database for testing, and our database user does not have enough rights to create another database on MySQL server. Ok, we can create an empty database in cPanel MySQL Databases and use the parameter --keepdb, but there is another problem: cPanel requires that database name starts with 'yournamexy_'. We cannot prepend 'test_' to the name.

Ok, there are still possibilities. Let's create a database with the name 'yournamexy_start_test'. This is allowed. How should we tell Django to use this name? We need to edit startapp/settings.py one more time and add an option to DATABASES section:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'yournamexy_start',
        'USER': 'yournamexy_start',
        'PASSWORD': os.environ['START_DB_PASSWORD'],
        'HOST': 'localhost',

        'TEST': {
            'NAME': 'yournamexy_start_test',
        }
    }
}

Now we can run tests with the following command:

python manage.py test --keepdb

(Don't even think of using the live db for testing - running tests on live will empty all data!)

Logging

Setting up logging in Django is really complicated. I don't want to go in detail here. But what can we do with the default configuration? Actually, it just works, although to use it is not very convenient. To demonstrate logging, let's add some logging to our model:

startapp/models.py

from django.db import models
import logging
logger = logging.getLogger(__name__)

# Create your models here.
class Contact(models.Model):
    forename = models.CharField(max_length=100)
    surname = models.CharField(max_length=100)
    email = models.EmailField(max_length=100)

    def __str__(self):
        return f"{self.forename} {self.surname}: {self.email}"

def save(self, *args, **kwargs):
    super().save(*args, **kwargs)  # Call the "real" save() method.    
    logger.error( f"Contact saved with id={self.id}.")

The default logging level is set to ERROR (despite whatever is written in Django documentation - probably Passenger or cPanel somehow interfere with it), so don't bother to use logger.debug of logger.info.

Now go to admin interface and create a few Contacts to see how it works.

Where can we see the logs?

Remember, there was a Passenger log file, which name we set up in SOFTWARE - Python Applications panel? (we called it /home/yourname/start.log). All server logs from our Python application go into this file, and logs generated by calling logger.error or logger.critical also go into this file. Not very convenient, but at least we can use the default Python logging mechanism.

Another useful option would be to install Django Debug Toolbar, although it still requires some configuration for optimal use.

Conclusion

In this tutorial, we have created a very basic Django application. This application does very little, and even does not have a proper front end, all it has is an admin part. But we discussed very important details, related to specifics of HostPresto! How to set up a Python environment, how to switch to MySQL database, how to set up passenger_wsgi.py file, how to use logging with minimum configuration effort and even how to test our application.

Probably, using a remote server as a development server is not a very good idea. It would be better to install Python and all development tools locally and only use the remote server as a test or production environment.

All files for this tutorial can be found here:

https://github.com/vallka/django-start

HostPresto! is reasonably priced and reliable hosting. What makes it outstanding — like many hosting providers they give you PHP and MySQL out of the box, but unlike most of other providers, they give you a very simple solution to host almost any kind of application, Node.js, Django, Flask, Ruby-on-Rails to mention a few. And all this zoo of applications can co-exist on the same server! What makes HostPresto! a great choice for any developer.


HostPresto are one of the leading independent, affordable website hosts.
hostpresto.com