Upgrade to Synnefo v0.13

The bulk of the upgrade to v0.13 is about user and quota migrations. In summary, the migration process has 3 steps:

  1. Run some commands and scripts to diagnose and extract some migration data while the OLD code is running, and BEFORE any changes are made.
  2. Bring down services, upgrade packages, configure services, and perform django database migrations. These migrations do not need any interaction between services.
  3. Initialize the Astakos quota system and bring the Astakos service up, since it will be needed during a second-phase of UUID and quota migrations, that also uses data extracted from step 1.

Warning

It is strongly suggested that you keep separate database backups for each service after the completion of each of step.

1. Bring web services down, backup databases

  1. All web services must be brought down so that the database maintains a predictable and consistent state during the migration process:

    # service gunicorn stop
    # service snf-dispatcher stop
    # etc.
    
  2. Backup databases for recovery to a pre-migration state.

  3. Keep the database servers running during the migration process

2. Prepare astakos user migration to case insensitive emails

It is possible that two or more users have been registered with emails that differ only in the case of its letters. There can only be one of those accounts after the migration, so the rest must be deleted.

Note that even if the users are deleted in Astakos, there still are duplicate entries in Cyclades and Pithos. For each service we need to reduce those multiple accounts into one, either merging them together, or deleting and discarding data from all but one.

2.1 Find duplicate email entries in Astakos

(script: find_astakos_users_with_conflicting_emails.py):

astakos-host$ cat << EOF > find_astakos_users_with_conflicting_emails.py
#!/usr/bin/env python
import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'

import astakos
from astakos.im.models import AstakosUser as A


def user_filter(user):
    return A.objects.filter(email__iexact=user.email).count() > 1

all_users = list(A.objects.all())
userlist = [(str(u.pk) + ': ' + str(u.email) + ' (' + str(u.is_active) + ', ' +
             str(u.date_joined) + ')') for u in filter(user_filter, all_users)]

sys.stderr.write("id email (is_active, creation date)\n")
print "\n".join(userlist)
EOF

astakos-host$ python ./find_astakos_users_with_conflicting_emails.py

2.1 Remove duplicate users in Astakos by their id

(script: delete_astakos_users.py):

astakos-host$ cat << EOF > delete_astakos_users.py
#!/usr/bin/env python

import os
import sys
from time import sleep

os.environ['DJANGO_SETTINGS_MODULE'] = 'synnefo.settings'

import astakos
from astakos.im.models import AstakosUser as A


def user_filter(user):
    return A.objects.filter(email__iexact=user.email).count() > 1

argv = sys.argv
argc = len(sys.argv)

if argc < 2:
    print "Usage: ./delete_astakos_users.py <id>..."
    raise SystemExit()

id_list = [int(x) for x in argv[1:]]

print ""
print "This will permanently delete the following users:\n"
print "id  email (is_active, creation date)"
print "--  --------------------------------"

users = A.objects.filter(id__in=id_list)
for user in users:
    print "%s: %s (%s, %s)" % (user.id, user.email, user.is_active,
                               user.date_joined)

print "\nExecute? (yes/no): ",
line = raw_input().rstrip()
if line != 'yes':
    print "\nCancelled"
    raise SystemExit()

print "\nConfirmed."
sleep(2)
for user in users:
    print "deleting %s: %s" % (user.id, user.email)
    user.delete()

EOF

astakos-host$ python ./delete_astakos_users.py 30 40

Warning

After deleting users with the delete_astakos_users.py script, check again with find_astakos_users_with_conflicting_emails.py (as in 2.1 Find duplicate email entries in Astakos) to make sure that no duplicate email conflicts remain.

3. Upgrade Synnefo and configure settings

3.1 Install the new versions of packages

astakos.host$ apt-get install \
                        kamaki \
                        snf-common \
                        snf-webproject \
                        snf-quotaholder-app \
                        snf-astakos-app \


cyclades.host$ apt-get install \
                        kamaki \
                        snf-common \
                        snf-webproject
                        snf-pithos-backend \
                        snf-cyclades-app \


pithos.host$ apt-get install \
                        kamaki \
                        snf-common \
                        snf-webproject
                        snf-pithos-backend \
                        snf-pithos-app \
                        snf-pithos-webclient \


ganeti.node$ apt-get install \
                        kamaki \
                        snf-common \
                        snf-cyclades-gtools \
                        snf-pithos-backend \

Note

Installing the packages will cause services to start. Make sure you bring them down again (at least gunicorn, snf-dispatcher)

3.2 Sync and migrate Django DB

Note

If you are asked about stale content types during the migration process, answer ‘no’ and let the migration finish.

astakos-host$ snf-manage syncdb
astakos-host$ snf-manage migrate

cyclades-host$ snf-manage syncdb
cyclades-host$ snf-manage migrate

Note

After the migration, Astakos has created uuids for all users, and has set the uuid as the public identifier of a user. This uuid is to be used both at other services (Cyclades, Pithos) and at the clientside (kamaki client settings).

Duplicate-email users have been deleted earlier in 2.1 Remove duplicate users in Astakos by their id

3.3 Setup quota settings for all services

Generally:

# Service       Setting                       Value
# quotaholder:  QUOTAHOLDER_URL            = https://quotaholder.host/quotaholder/v
# quotaholder:  QUOTAHOLDER_TOKEN          = <random string>

# astakos:      ASTAKOS_QUOTAHOLDER_TOKEN  = <the same random string>
# astakos:      ASTAKOS_QUOTAHOLDER_URL    = https://quotaholder.host/quotaholder/v

# cyclades:     CYCLADES_QUOTAHOLDER_TOKEN = <the same random string>
# cyclades:     CYCLADES_QUOTAHOLDER_URL   = https://quotaholder.host/quotaholder/v
# cyclades:     CYCLADES_USE_QUOTAHOLDER   = True


# pithos:       PITHOS_QUOTAHOLDER_TOKEN   = <the same random string>
# pithos:       PITHOS_QUOTAHOLDER_URL     = https://quotaholder.host/quotaholder/v
# pithos:       PITHOS_USE_QUOTAHOLDER     = True
# All services must match the quotaholder token and url configured for quotaholder.

Specifically:

On the Astakos host, edit /etc/synnefo/20-snf-astakos-app-settings.conf:

QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
ASTAKOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
ASTAKOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'

On the Cyclades host, edit /etc/synnefo/20-snf-cyclades-app-quotas.conf:

CYCLADES_USE_QUOTAHOLDER = True
CYCLADES_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
CYCLADES_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'

# Set to False if astakos & cyclades are on the same host
#CYCLADES_PROXY_USER_SERVICES = True

Note

If Cylcades and Astakos are installed on the same server, set CYCLADES_PROXY_USER_SERVICES = False

On the Pithos host, edit /etc/synnefo/20-snf-pithos-app-settings.conf:

PITHOS_QUOTAHOLDER_URL = 'https://accounts.example.synnefo.org/quotaholder/v'
PITHOS_QUOTAHOLDER_TOKEN = 'aExampleTokenJbFm12w'
PITHOS_USE_QUOTAHOLDER = False # will set to True after migration

Note

During the migration it must be set, PITHOS_USE_QUOTAHOLDER = False. Set to True once the migration is over.

3.4 Setup astakos

  • Remove this redirection from astakos front-end web server

        RewriteRule ^/login(.*) /im/login/redirect$1 [PT,NE]
    
    (see `<http://docs.dev.grnet.gr/synnefo/latest/quick-install-admin-guide.html#apache2-setup>`_)
  • Enable users to change their contact email. Edit

/etc/synnefo/20-snf-astakos-app-settings.conf

ASTAKOS_EMAILCHANGE_ENABLED = True
  • Rename the following (Astakos-specific) setting:

    ASTAKOS_DEFAULT_FROM_EMAIL
    

    to this (Django-specific) name:

    SERVER_EMAIL
    
  • Instead of using the following (Astakos-specific) setting:

    ASTAKOS_DEFAULT_ADMIN_EMAIL
    

    include one or more entries in this (Django-specific) setting:

    ADMINS = (
        ('Joe Doe', 'doe@example.net'),
        ('Mary Jean', 'mary@example.net'),
    )
    

Note

The SERVER_EMAIL and ADMINS settings are Django-specific. As such they will be the shared for any two (or more) services that happen to be collocated within the same application server (e.g. astakos & cyclades within the same gunicorn)

3.5 Setup Cyclades

  • Run on the Astakos host

    # snf-manage service-list
    
  • Set the Cyclades service token in /etc/synnefo/20-snf-cyclades-app-api.conf

    CYCLADES_ASTAKOS_SERVICE_TOKEN = 'asfasdf_CycladesServiceToken_iknl'
    
  • Since version 0.13, Synnefo uses VMAPI in order to prevent sensitive data needed by ‘snf-image’ to be stored in Ganeti configuration (e.g. VM password). This is achieved by storing all sensitive information to a CACHE backend and exporting it via VMAPI. The cache entries are invalidated after the first request. Synnefo uses memcached as a django cache backend. To install, run on the Cyclades host:

    apt-get install memcached
    apt-get install python-memcache

    You will also need to configure Cyclades to use the memcached cache backend. Namely, you need to set IP address and port of the memcached daemon, and the default timeout (seconds tha value is stored in the cache). Edit /etc/synnefo/20-snf-cyclades-app-vmapi.conf

    VMAPI_CACHE_BACKEND = "memcached://127.0.0.1:11211/?timeout=3600"
    

    Finally, set the BASE_URL for the VMAPI, which is actually the base URL of Cyclades, again in /etc/synnefo/20-snf-cyclades-app-vmapi.conf. Make sure the domain is exaclty the same, so that no re-directs happen

    VMAPI_BASE_URL = "https://cyclades.example.synnefo.org"
    

    Note

    • These settings are needed in all Cyclades workers.
    • VMAPI_CACHE_BACKEND just overrides django’s CACHE_BACKEND setting
    • memcached must be reachable from all Cyclades workers.
    • For more information about configuring django to use memcached: https://docs.djangoproject.com/en/1.2/topics/cache

3.6 Setup Pithos

  • Pithos forwards user catalog services to Astakos so that web clients may access them for uuid-displayname translations. Edit on the Pithos host /etc/synnefo/20-snf-pithos-app-settings.conf

    PITHOS_USER_CATALOG_URL    = https://accounts.example.synnefo.org/user_catalogs/
    PITHOS_USER_FEEDBACK_URL   = https://accounts.example.synnefo.org/feedback/
    PITHOS_USER_LOGIN_URL      = https://accounts.example.synnefo.org/login/
    
    # Set to False if astakos & pithos are on the same host
    #PITHOS_PROXY_USER_SERVICES = True

Note

If Pithos and Astakos are installed on the same server, set PITHOS_PROXY_USER_SERVICES = False

4. Start astakos and quota services

Warning

To ensure consistency, prevent public access to astakos during migrations. This can be done via firewall or webserver access control.

Start (or restart, if running) the webserver and gunicorn on the Astakos host. E.g.:

# service apache2 start
# service gunicorn start

5. Load resource definitions into Astakos

First, set the corresponding values on the following dict in /etc/synnefo/20-snf-astakos-app-settings.conf

# Set the cloud service properties
ASTAKOS_SERVICES = {
    'cyclades': {
#        # Specifying the key 'url' will overwrite it.
#        # Use this to (re)set service URL.
#        'url': 'https://cyclades.example.synnefo.org/ui/',
#        # order services in listings, cloudbar, etc.
#        'order' : 1
        'resources': [{
            'name': 'disk',
            'group': 'compute',
            'uplimit': 30*1024*1024*1024,
            'unit': 'bytes',
            'desc': 'Virtual machine disk size'
            }, {
            'name': 'cpu',
            'group': 'compute',
            'uplimit': 6,
            'desc': 'Number of virtual machine processors'
            }, {
            'name': 'ram',
            'group': 'compute',
            'uplimit': 6*1024*1024*1024,
            'unit': 'bytes',
            'desc': 'Virtual machines'
            }, {
            'name': 'vm',
            'group': 'compute',
            'uplimit': 2,
            'desc': 'Number of virtual machines'
            }, {
            'name': 'network.private',
            'group': 'network',
            'uplimit': 1,
            'desc': 'Private networks'
            }
        ]
    },
    'pithos+': {
#        # Use this to (re)set service URL.
#        'url': 'https://pithos.example.synnefo.org/ui/',
#        # order services in listings, cloudbar, etc.
#        'order' : 2
        'resources':[{
            'name': 'diskspace',
            'group': 'storage',
            'uplimit': 5*1024*1024*1024,
            'unit': 'bytes',
            'desc': 'Pithos account diskspace'
            }]
    }
}

Note

The name of the Pithos service is pithos+. If you have named your pithos service pithos, without +, then you must rename it:

$ snf-manage service-list | grep pithos # find service id
$ snf-manage service-update --name='pithos+' <service id>

Then, configure and load the available resources per service and associated default limits into Astakos. On the Astakos host run

# snf-manage astakos-init --load-service-resources

Note

Before v0.13, only cyclades.vm, cyclades.network.private, and pithos+.diskspace existed (not with these names, there were per-service settings). However, limits to the new resources must also be set.

If the intetion is to keep a resource unlimited, (counting on that VM creation will be limited by other resources’ limit) it is best to calculate a value that is too large to be reached because of other limits (and available flavours), but not much larger than needed because this might confuse users who do not readily understand that multiple limits apply and flavors are limited.

6. Migrate Services user names to uuids

6.1 Double-check cyclades before user case/uuid migration

cyclades.host$ snf-manage cyclades-astakos-migrate-013 --validate

Duplicate user found?

  • either merge (merge will merge all resources to one user):

    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --merge-user=kpap@grnet.gr
  • or delete

    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=KPap@grnet.gr
    # (only KPap will be deleted not kpap)

6.2 Double-check pithos before user case/uuid migration

pithos.host$ snf-manage pithos-manage-accounts --list-duplicate

Duplicate user found?

If you want to migrate files first:

  • merge (merge will merge all resources to one user):

    pithos.host$ snf-manage pithos-manage-accounts --merge-accounts --src-account=SPapagian@grnet.gr --dest-account=spapagian@grnet.gr
    # (SPapagian@grnet.gr's contents will be merged into spapagian@grnet.gr, but SPapagian@grnet.gr account will still exist)
  • and then delete

    pithos.host$ snf-manage pithos-manage-accounts --delete-account=SPapagian@grnet.gr
    # (only SPapagian@grnet.gr will be deleted not spapagian@grnet.gr)

If you do NOT want to migrate files just run the second step and delete the duplicate account.

6.3 Migrate Cyclades users (email case/uuid)

cyclades.host$ snf-manage cyclades-astakos-migrate-013 --migrate-users
  • if invalid usernames are found, verify that they do not exist in astakos:

    astakos.host$ snf-manage user-list
  • if no user exists:

    cyclades.host$ snf-manage cyclades-astakos-migrate-013 --delete-user=<userid>

Finally, if you have set manually quotas for specific users inside /etc/synnefo/20-snf-cyclades-app-api.conf (in VMS_USER_QUOTA, NETWORKS_USER_QUOTA make sure to update them so that:

  1. There are no double entries wrt case sensitivity
  2. Replace all user email addresses with the corresponding UUIDs

To find the UUIDs for step 2 run on the Astakos host

# snf-manage user-list

6.4 Migrate Pithos user names

Check if alembic has not been initialized

pithos.host$ pithos-migrate current
  • If alembic current is None (e.g. okeanos.io)

    pithos.host$ pithos-migrate stamp 3dd56e750a3

Then, migrate pithos account name to uuid:

pithos.host$ pithos-migrate upgrade head

Finally, set this setting to True:

PITHOS_USE_QUOTAHOLDER = True

7. Migrate old quota limits

7.1 Migrate Pithos quota limits to Astakos

Migrate from pithos native to astakos/quotaholder. This requires a file to be transfered from Cyclades to Astakos:

pithos.host$ snf-manage pithos-export-quota --location=pithos-quota.txt
pithos.host$ scp pithos-quota.txt astakos.host:
astakos.host$ snf-manage user-set-initial-quota pithos-quota.txt

Note

pithos-export-quota will only export quotas that are not equal to the defaults in Pithos. Therefore, it is possible to both change or maintain the default quotas across the migration. To maintain quotas the new default pithos+.diskpace limit in Astakos must be equal to the (old) default quota limit in Pithos. Change either one of them make them equal.

see 5. Load resource definitions into Astakos on how to set the (new) default quotas in Astakos.

7.2 Migrate Cyclades quota limits to Astakos

cyclades.host$ snf-manage cyclades-export-quota --location=cyclades-quota.txt
cyclades.host$ scp cyclades-quota.txt astakos.host:
astakos.host$ snf-manage user-set-initial-quota cyclades-quota.txt

cyclades-export-quota will only export quotas that are not equal to the defaults. See note above.

8. Enforce the new quota limits migrated to Astakos

The following should report all users not having quota limits set because the effective quota database has not been initialized yet.

astakos.host$ snf-manage astakos-quota --verify

Initialize the effective quota database:

astakos.host$ snf-manage astakos-quota --sync

This procedure may be used to verify and re-synchronize the effective quota database with the quota limits that are derived from policies in Astakos (initial quotas, project memberships, etc.)

9. Initialize resource usage

The effective quota database (quotaholder) has just been initialized and knows nothing of the current resource usage. Therefore, each service must send it in.

9.1 Initialize Pithos resource usage

pithos.host$ snf-manage pithos-reset-usage

9.2 Initialize Cyclades resource usage

cyclades.host$ snf-manage cyclades-reset-usage

10. Install periodic project maintainance checks

In order to detect and effect project expiration, a management command has to be run periodically (depending on the required granularity, e.g. once a day/hour):

astakos.host$ snf-manage project-control --terminate-expired

A list of expired projects can be extracted with:

astakos.host$ snf-manage project-control --list-expired

11. Restart all services

Start (or restart, if running) all Synnefo services on all hosts.

# service gunicorn restart
# service snf-dispatcher restart
# etc.