python - @kholidfu

Table of Contents

1 Deploy Flask + Nginx + Supervisor di Ubuntu

1.1 Add user di Ubuntu

  adduser sopier
  adduser sopier sudo

1.2 Install needed package

  sudo apt-get install build-essential python-dev python-pip nginx
  emacs24-nox git

1.3 Install virtualenv & supervisor

  sudo pip install virtualenv supervisor

1.4 git clone

  git clone https://github.com/sopier/example.com

1.5 Setting virtualenv

  cd example.com
  virtualenv .
  . bin/activate
  pip install -r requirements.txt
  pip install uwsgi

1.6 Setting supervisor

Buat berkas baru, misal di ~/supervisord.conf, dan isikan baris konfigurasi berikut:

  [unix_http_server]
  file=/tmp/uwsgi.sock   ; (the path to the socket file)

  [supervisord]
  logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
  logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
  logfile_backups=10          ; (num of main logfile rotation backups;default 10)
  loglevel=info               ; (log level;default info; others: debug,warn,trace)
  pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
  nodaemon=false              ; (start in foreground if true;default false)
  minfds=1024                 ; (min. avail startup file descriptors;default 1024)
  minprocs=200

  [rpcinterface:supervisor]
  supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

  [supervisorctl]
  logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
  serverurl=unix:///tmp/uwsgi.sock ; use a unix:// URL

  [program:example.com]
  command=/home/sopier/example.com/bin/uwsgi 
              --socket /tmp/uwsgi.sock 
              --module run 
              --callable app -H /home/sopier/example.com/ 
              --chdir /home/sopier/example.com/ 
              --chmod-socket=666 
              --processes=5
  directory=/home/sopier/example.com/
  autostart=true
  autorestart=true
  stdout_logfile=/tmp/example.log
  redirect_stderr=true
  stopsignal=QUIT

1.7 Sunting berkas run.py

Berikut ini setting untuk run.py (sesuaikan seperlunya):

  #!/usr/bin/env python
  from app import app

  if __name__ == "__main__":
      app.run(debug=False)

1.8 Setting nginx

Berikut ini adalah konfigurasi nginx sederhana tapi works:

  server {
      listen        80;
      server_name   www.example.com;
      location / { try_files $uri @app; }
      location @app {
          include uwsgi_params;
          uwsgi_pass unix:/tmp/uwsgi.sock;
      }
  }

Atau jika Anda ingin multiple site, berikut ini contoh konfigurasinya:

  server {
      listen        80;
      server_name   www.example1.com;
      location / { try_files $uri @app; }
      location @app {
          include uwsgi_params;
          uwsgi_pass unix:/tmp/uwsgi1.sock;
      }
  }
  server {
      listen        80;
      server_name   www.example2.com;
      location / { try_files $uri @app; }
      location @app {
          include uwsgi_params;
          uwsgi_pass unix:/tmp/uwsgi2.sock;
      }
  }

1.9 Jalankan supervisor

Terakhir, jalankan supervisor dengan menjalankan perintah berikut:

  sudo supervisord -c ~/supervisord.conf

2 Install library lxml di dalam virtualenv

  sudo apt-get install libxml2-dev libxslt-dev

Jika server Anda terlalu kecil spec-nya (gagal di DO yang $5/month, bisa jadi terjadi error, ini terjadi karena proses compile membutuhkan memory besar. Untuk menambah memory, kita sementara dapat menambah swap.

Caranya adalah sebagai berikut:

# create new partition
dd if=/dev/zero of=/swapfile bs=1M count=1024
# set as swap
mkswap /swapfile
# activate the swap once
swapon /swapfile
# run every machine reboot (optional)
/swapfile swap swap defaults 0 0 # add into /etc/fstab

3 Catch search engine terms with python, flask and mongo

  # pip install referer_parser
  from referer_parser import Referer
  # dbase
  import pymongo

  c = pymongo.Connection()
  refdb = c['referer']

  # ganti "-" => " " pada q
  t = q.replace("-", " ")

  # referer tracker
  try:
      head = request.headers
      url = head['Referer']
      refobj = Referer(url)
      # setem ini sudah bersih!
      setem = refobj.search_term

      # jika ada setem, input into db                                                

      if setem:
          # jika lom ada setem, insert!                                              

          if refdb.term.find_one({"q": t}) is None:
              refdb.term.insert({"q": t, "refer": {setem: 1}})
          # jika sudah ada, pilihannya ada 2, set atau increment!
          else:
              # jika sudah ada setem, increment!                                    

              if setem in refdb.term.find_one({"q": t})['refer']:
                  refdb.term.update(
                      {"q": t},
                      {"$inc": {"refer." + str(setem): 1}},
                      upsert=True
                  )
              # jika lom ada setem, set setem: 1!                                    

              elif setem not in refdb.term.find_one({"q": t})['refer']:
                  refdb.term.update(
                      {"q": t},
                      {"$set": {"refer." + str(setem): 1}},
                      upsert=True
                  )
  except:
      pass

  # sorting term
  sorted(
      db.term.find_one({"q": t})['refer'].items(),
      key=lambda x: x[1],
      reverse=True
  )

4 Mencegah Cascade di django models

django, by default menggunakan cascade untuk model ForeignKey, ini artinya jika data child dihapus maka data parent juga ikut terhapus. Untuk mencegah hal ini, kita dapat menambahkan argument berikut:

  nama = models.ForeignKey(Guru, on_delete=models.SET_NULL)

Jika setting sudah benar, tandanya ada tanda silang di sebelah field ForeignKey.

5 CSS, JS dan IMG tidak mau load di django-admin

Jika hal ini terjadi, berarti setting nginx Anda perlu dibenahi, yakni dengan menambahkan location static di konfig nginx Anda, misal sebagai berikut:

  server {
      ...
  
      location /static/ {
          alias /home/sopier/mslib/mslib/static/;
      }
      
      ...
  }

6 Setting gunicorn + django + nginx

Dengan asumsi struktur direktori sebagai berikut:

  mslib/
  ├── app
  │   ├── admin.py
  │   ├── __init__.py
  │   ├── migrations
  │   ├── models.py
  │   ├── tests.py
  │   ├── urls.py
  │   ├── views.py
  ├── manage.py
  ├── mslib
  │   ├── __init__.py
  │   ├── settings.py
  │   ├── urls.py
  │   ├── wsgi.py
  ├── static
  │   ├── admin
  │   ├── app
  │   └── js
  └── templates
      ├── 404.html
      ├── admin
      └── app

Dan berikut ini setting untuk nginx:

  upstream app_server {
     server 127.0.0.1:8000 fail_timeout=0;
  }
  
  server {
     listen 80;
     server_name 159.xxx.xxx.xxx;
     client_max_body_size 4G;
     proxy_read_timeout 1200;
  
     location / {
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header Host $http_host;
         proxy_redirect off;
         proxy_pass http://app_server;
         if (!-f $request_filename) {
             proxy_pass http://app_server;
             break;
         }
     }
  
     location /static/ {
         alias /home/sopier/mslib/mslib/static/;
     }
  
     location /media/ {
         alias /home/sopier/mslib/mslib/collections/;
     }
  }

Maka untuk menjalankan gunicorn:

  gunicorn --env DJANGO_SETTINGS_MODULE=mslib.settings mslib.wsgi --bind 127.0.0.1:8000

7 Up and Running with django-cookiecutter

# starting django project with cookiecutter

cookiecutter https://github.com/pydanny/cookiecutter-django.git

# choose Y
# It will clone in ~/.cookiecutter
# Follow the wizard

cd into project_name
virtualenv .

# edit requirements/base.txt, add/comment unwanted libs
pip install -r requirements/local.txt

# install gunicorn
pip install gunicorn

# create database name 
sudo -u postgres -i
createdb project_name

# do migration
python manage.py migrate
python manage.py runserver

# creating project/apps
# you can create app straightforward in project_name/ dir
python manage.py createapp myapp

# or
# startproject and createapp
python manage.py startproject myproject
cd myproject
python manage.py createapp myapp

# I prefer the 1st one

# run server with gunicorn
gunicorn --env DJANGO_SETTINGS_MODULE=config.settings.local config.wsgi --bind 127.0.0.1:8000

Selanjutnya tinggal buat model di django seperti biasa.

8 Akses ebay SOAP API dengan Python

This is how accessing soap api xml in python.

import json
import requests
from ebaysdk.response import ResponseDataObject, Response

def get_countries():
    """Request SOAP API for Country.
    """

    headers = {
        'X-EBAY-API-SITEID': 0,
        'X-EBAY-API-DEV-NAME': 'your-api-dev',
        'X-EBAY-API-CERT-NAME': 'your-api-cert',
        'Content-Length': '1096',
        'X-EBAY-API-APP-NAME': 'your-api-app',
        'X-EBAY-API-COMPATIBILITY-LEVEL': '837',
        'X-EBAY-SDK-REQUEST-ID': 'your-sdk-request',
        'X-EBAY-API-CALL-NAME': u'GetEbayDetails',
        'User-Agent': 'eBaySDK/2.1.2 Python/2.7.6 Linux/3.13.0-37-generic',
        'Content-Type': 'text/xml'
    }

    body = """
<?xml version="1.0" encoding="utf-8"?>
<GeteBayDetailsRequest xmlns="urn:ebay:apis:eBLBaseComponents">
  <RequesterCredentials>
    <eBayAuthToken>your-token</eBayAuthToken>
  </RequesterCredentials>
  <DetailName>CountryDetails</DetailName>
</GeteBayDetailsRequest>
"""
    resp = requests.post('https://api.ebay.com/ws/api.dll', data=body, headers=headers)
    from ebaysdk.response import ResponseDataObject, Response
    xml = resp.content

    o = ResponseDataObject({'content': xml}, [])
    r = Response(o, verb='findItemsByProduct', list_nodes=['searchResult.item'])

    data = json.loads(r.json(), 'utf8')
    countries = [(i['Country'], i['Description']) for i in data['GeteBayDetailsResponse']['CountryDetails']]
    return countries

9 Konfigurasi dan Implementasi PostgreSQL HStoreField di django

Install postgresql-contrib-9.3:

sudo apt-get install postgresql-contrib-9.3

Kemudian jalankan postgre console

sudo -u postgres -i
psql
\connect dbname
create extension hstore;

Output:

CREATE EXTENSION

Kemudian deskripsikan model di django:

from django.contrib.postgres.fields import HStoreField

class Item(models.Model):
    itemId = models.CharField(max_length=50)
    title = models.CharField(max_length=1000)
    sellerInfo = HStoreField()

    def __unicode__(self):
        return self.title

Contoh query:

i = Item(name='kholidfu', data={'skill': 'python'})
i.save()

10 Setting nginx + uwsgi + supervisor + django

Berikut setting nginx:

server {
      listen        80;
      server_name   www.example.com;
      location / { try_files $uri @app; }
      location @app {
          include uwsgi_params;
          uwsgi_pass unix:/tmp/uwsgi.sock;
      }
}

Setting uwsgi di supervisord.conf:

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200

[program:kresna]
command=/home/sopier/kresna2/bin/uwsgi --socket /tmp/uwsgi.sock --module=config.wsgi:application --env DJANGO_SETTINGS_MODULE=config.settings.production --callable app -H /home/sopier/kresna2/ --chdir /home/sopier/kresna2/ --chmod-socket=666 --processes=5
directory=/home/sopier/kresna2/
autostart=true
autorestart=true
stdout_logfile=/tmp/kresna2.log
redirect_stderr=true

Yang harus Anda perhatikan disini adalah letak dari module wsgi, karena saya di sini menggunakan cookiecutter dari pydanny, maka secara default, file wsgi.py berada di direktori config. Kalau menggunakan versi default startproject dari django, file wsgi.py berada di direktori myproject.wsgi.

Selamat mencoba!

11 Port di django vagrant

Jika menggunakan vagrant, setelah VagrantFile terkonfigurasi dengan baik dan benar, agar dapat diakses oleh OS Host, jalankan perintah berikut:

  ./manage.py runserver [::]:8000

Jangan lupa, sesuaikan port number dengan VagrantFile Anda!

Date: 2015-10-01

Author: Kholid Fuadi

Created: 2016-10-14 Fri 11:36

Validate