Tag: devops

  • My First Wrong After The Vacation

    At the end of my three-week vacation, I was asked for a seemingly easy task. We were going to release the new version of the project we developed:

    • Is today is the last working of the week? No.
    • Is there any problem with the unit tests? No.
    • Are all changes confirmed by the product manager, in the staging server? Yes.

    I always note everything, and so I write this article to remind this experience. My other habit is to define dosh sub-commands to access frequently used commands in my whole projects. It doesn’t matter if the project is written in .NET Core or used Django;

    • I know that the project will start in my development environment with dosh start,
    • I know that the project will build with dosh build,
    • I know that dosh deploy command will be running if I push my commits to some specific git branches. All these things are common sub-commands in my projects.

    So what was not a standard?

    I manage everything in a project except the server maintenance. Today the site was offline about an hour because of the communication failure. Sometimes there are small but critical upgrades on the servers, and one of them broke SSL module of the project’s psycopg2 dependency. I had only these logs:

    DAMN ! worker 1 (pid: 28018) died :( trying respawn ...
    Respawned uWSGI worker 1 (new pid: 28021)
    !!! uWSGI process 27988 got Segmentation Fault !!!
    *** backtrace of 27988 ***
    /usr/bin/uwsgi-core(uwsgi_backtrace+0x2a) [0x55a0ad5c498a]
    /usr/bin/uwsgi-core(uwsgi_segfault+0x23) [0x55a0ad5c4d73]
    /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f4811ecaf20]
    /project/venv/lib/python3.6/site-packages/psycopg2/.libs/./libssl-8bb9b3dd.so.1.0.2o(ssl3_cleanup_key_block+0xb) [0x7f480a747cbb]
    /project/venv/lib/python3.6/site-packages/psycopg2/.libs/./libssl-8bb9b3dd.so.1.0.2o(ssl3_clear+0x16) [0x7f480a7455b6]
    /project/venv/lib/python3.6/site-packages/psycopg2/.libs/./libssl-8bb9b3dd.so.1.0.2o(tls1_clear+0x9) [0x7f480a751229]
    /usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_new+0x3f2) [0x7f4812b429d2]
    /project/venv/lib/python3.6/site-packages/psycopg2/.libs/libpq-239f9ac8.so.5.10(+0x23399) [0x7f480a9dc399]
    

    You may know Segmentation Fault if you coded in C or if you worked with a package manager. Some included libraries that are coming with psycopg2 binary package are linked to the old version of libssl, so the last server update broke the dependencies1.

    lib/python3.6/site-packages/psycopg2/.libs/libssl-8bb9b3dd.so.1.0.2o   # from 1.0
    lib/python3.6/site-packages/psycopg2/.libs/libssl-7a80b951.so.1.1.1b   # to 1.1
    

    In some projects, it’s not possible to solve the problems as we wanted because of some regulations and source limits. I could find prevention for that case, but somehow, I forgot to check the updates in the source code and server packages and tried to make the deployment. And the worst thing, I didn’t realise the problem until someone warned me.

    So, I added one more rule for myself to make a deployment:

    • Is today the first day after the vacation? Yes (so, don’t touch it.).

    1. “Don’t break the dependencies”, I missed contributing a Linux distro like Pardus. ↩︎
  • How do I publish my articles?

    With just a command:

    # terminal
    $ ENV=PROD ./do.sh deploy

    I have many projects using my script named DOSH and the parameters are mostly in common; deploystartstopruntests, etc. It can be a .NET Core or a Django project, or just a website using any static site generator like this one but I run all the projects with the same command: dosh start.

    In Windows, the command may be a bit longer than in the other operating systems. So it’s a good idea to shorten the command using some aliases, the default environment is DEV:

    # ~/.bash_profile
    alias dosh="winpty bash do.sh"  # or just type "./do.sh" in Linux or MacOS.
    # terminal
    $ dosh start  # in DEV
    $ ENV=PROD dosh deploy  # in PROD, my deploy parameter works only in production.

    Do you use PowerShell? No problem, I think it’s better to do the same thing in PowerShell, you can find the sample codes below, but let me show you how I use that, it’s a bit different than BASH:

    # powershell
    ~\Workspace\website [master ≡]> $env:ENV = "PROD"
    ~\Workspace\website [master ≡]> dosh deploy
    Environment: production
    Deploying updates to GitHub...
    Building sites …
                       | EN | TR
    +------------------+----+----+
      Pages            | 13 | 67
      Paginator pages  |  0 |  0
      Non-page files   |  0 |  0
      Static files     | 63 | 63
      Processed images |  0 |  0
      Aliases          |  1 | 12
      Sitemaps         |  2 |  1
      Cleaned          |  0 |  0
    
    Total in 748 ms
    [master 1686121] rebuilding site 12/30/2018 15:02:32
     1 file changed, 3 insertions(+), 3 deletions(-)
    Enumerating objects: 9, done.
    Counting objects: 100% (9/9), done.
    Delta compression using up to 12 threads
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (5/5), 628 bytes | 628.00 KiB/s, done.
    Total 5 (delta 3), reused 0 (delta 0)
    remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
    To github.com:gkmngrgn/gkmngrgn.github.io.git
       733793b..1686121  master -> master
    DONE.

    Code samples

    You can find many sample codes here to write a deployment script and I didn’t think of this idea first but still, I’m sharing my scripts to be an alternative:

    ./do.sh

    #!/usr/bin/env bash
    
    BLUE="\\033[1;34m"
    GREEN="\\033[1;32m"
    NORMAL="\\033[0;39m"
    RED="\\033[1;31m"
    
    print_help() {
        echo -e "${BLUE}Available environments${NORMAL}"
        echo "  - DEV (default)"
        echo "  - PROD"
        echo "  - STAGING"
        echo ""
        echo -e "${BLUE}Available commands${NORMAL}"
        echo -e "  > deploy [:message]      Deploy the changes"
        echo -e "  > start                  Run development server"
    }
    
    deploy() {
        case ${ENV} in
            "PROD")
                echo -e "${BLUE}Deploying updates to GitHub...${NORMAL}"
                ;;
            *)
                echo -e "${RED}This command only works in production.${NORMAL}"
                exit
                ;;
        esac
    
        if [ $# -eq 1 ]
        then msg="${1}"
        else msg="rebuilding site `date`"
        fi
    
        hugo
        cd public
        git add .
        git commit -m "${msg}"
        git push origin master
        cd ..
        echo -e "${GREEN}DONE.${NORMAL}"
    }
    
    start() {
        hugo server -D
    }
    
    if [ -z "$1" ]
    then
        print_help
    else
        case $ENV in
            "PROD")
                env_str="production"
                ;;
            "STAGING")
                env_str="staging"
                ;;
            *)
                env_str="development"
                ;;
        esac
        echo -e "${BLUE}Environment:${NORMAL} $env_str"
        "$@"
    fi

    ./do.ps1

    <# Variables #>
    if ($env:ENV -eq "PROD") {
        $env_str = "production"
    }
    elseif ($env:ENV -eq "STAGING") {
        $env_str = "staging"
    }
    else {
        if ($env:ENV -ne "DEV") {
            $env:ENV = "DEV"
        }
        $env_str = "development"
    }
    
    <# Functions #>
    function Get-Help {
        Write-Host -ForegroundColor Cyan "Available Environments"
        Write-Output "  - DEV (default)"
        Write-Output "  - PROD"
        Write-Output "  - STAGING"
        Write-Output ""
    
        Write-Host -ForegroundColor Cyan "Available commands"
        Write-Output "  > deploy [:message]      Deploy the changes"
        Write-Output "  > start                  Run development server"
    }
    
    function Start-Deploy ([string] $subParams) {
        if ($env:ENV -eq "PROD") {
            Write-Host -ForegroundColor Blue "Deploying updates to GitHub..."
        }
        else {
            Write-Host -ForegroundColor Red "This command only works in production."
            exit
        }
    
        if ($subParams -eq "") {
            $msg = "rebuilding site $(Get-Date)"
        } else {
            $msg = $subParams
        }
    
        Invoke-Expression "hugo"
        Invoke-Expression "cd public"
        Invoke-Expression "git add ."
        Invoke-Expression "git commit -m '$msg'"
        Invoke-Expression "git push origin master"
        Invoke-Expression "cd .."
        Write-Host -ForegroundColor Green "DONE."
    }
    
    function Start-Server () {
        Invoke-Expression "hugo server -D"
    }
    
    <# Set default parameter as help. #>
    if ($args.Length -eq 0) {
        $param = "help"
        $subParams = ""
    }
    else {
        $param, $subParams = $args
    }
    
    if ($param -ne "help") {
        Write-Host -ForegroundColor Cyan -NoNewLine "Environment: "
        Write-Output $env_str
    }
    
    <# We use switch instead of alias because there are some constant variables like `start`. #>
    switch ($param) {
        "deploy" { Start-Deploy $subParams }
        "start" { Start-Server }
        Default { Get-Help }
    }

    Don’t forget to set the alias in your config file, for example:

    # ~\Documents\PowerShell\profile.ps1
    $env:ENV = "DEV"
    Set-Alias dosh .\do.ps1