8.5. Clients reference

Kamaki library API consists of clients corresponding to the Synnefo API, which is equivalent to the OpenStack API with some extensions. In some cases, kamaki implements the corresponding OpenStack libraries as separate clients and the Synnefo extensions as class extensions of the former.

The kamaki library API consists of the following clients:

In kamaki.clients.astakos:

AstakosClient           An Identity and Account client for Synnefo API
OriginalAstakosClient   The client of the Synnefo astakosclient package
LoggedAstakosClient     The original client with kamaki-style logging
CachedAstakosClient     Some calls are cached to speed things up

Note

Use AstakosClient if you are not sure

TO BE COMPLETED

8.5.1. Astakos / Identity

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/identity-api-guide.html

The core functionality of this module is to authenticate a user and provide user data (e.g., email, unique user id)

8.5.1.1. Authenticate user

Example: Authenticate user, get name and uuid

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from kamaki.clients.astakos import AstakosClient, ClientError

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

try:
    user = astakos.authenticate()
except ClientError as ce:
    if ce in (401, ):
        print "Authentication failed, {0}".format(ce)
        exit(0)
    raise
user_info = user['access']['user']
print "Authentication was successful for {name}, with uuid {uuid}".format(
    name=user_info['name'], uuid=user_info['id'])

Note

the authenticate method returns a dict, which is defined by the Synnefo API (not by kamaki)

8.5.2. Astakos / Resources and Quotas

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/api-guide.html#resource-and-quota-service-api-astakos

This API provides information on available resources, resource usage and quota limits.

8.5.2.1. Resource quotas

Example: Resource usage and limits for number of VMs and IPs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from kamaki.clients.astakos import AstakosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']

#  Get resources assigned on my personal (system) project
all_resources = astakos.get_quotas()
my_resources = all_resources[uuid]

#  Print usage and limits for VMs and IPs
vms = my_resources["cyclades.vm"]
ips = my_resources["cyclades.floating_ip"]

print "You are using {vms}/{vm_limit} VMs and {ips}/{ip_limit} IPs".format(
    vms=vms["usage"], vm_limit=vms["limit"],
    ips=ips["usage"], ip_limit=ips["limit"])

Note

Quotas are defined by projects (see next section). Every user is member to a personal project (the “system” project) which is identified by the uuid of the user, but they may draw resources from other projects as well. In this script we only got the quota information related to the system project and we did that with this line of code my_resources = all_resources[uuid]

8.5.3. Astakos / Projects

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/api-guide.html#project-service-api

The relation between projects, users and resources:

cloud resources: VMs, CPUs, RAM, Volumes, IPs, VPNs, Storage space
a cloud user --- is member to --- projects
a cloud resource --- must be registered to --- a project
A user creates a resource: registers a resource to a project he is member of

What information is found in a project:

  • members: cloud users who can use the project resources
  • project limit: usage limits per resource for the whole project
  • member limit: usage limits per resource per cloud user
  • usage: current usage per resource per cloud user

Note

By default, every user has a personal (system) project. By default when a user creates a resource, it is registered to this project, except if they explicitly request to register a resource to another project.

8.5.3.1. Query my projects

Example: Get information for all projects I am member to

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from kamaki.clients.astakos import AstakosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']

#  Get all projects information
projects = astakos.get_projects()

for project in projects:
    print "Project {name} ({uuid})\n\t{description}\n\t{url}".format(
        name=project["name"], uuid=project["id"],
        description=project["description"], url=project["homepage"])

The results should look like this:

system:a1234567-a890-1234-56ae-78f90bb1c2db (a1234567-a890-1234-56ae-78f90bb1c2db)
    System project for user user@example.com

CS333 lab assignments (a9f87654-3af2-1e09-8765-43a2df1098765)
    Virtual clusters for CS333 assignments
    https://university.example.com/courses/cs333

8.5.3.2. Quotas per project

Example: Get usage and total limits per resource per project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from kamaki.clients.astakos import AstakosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']

#  Get resources assigned on my personal (system) project
all_resources = astakos.get_quotas()

for project, resources in all_resources.items():
    print "For project with id {project_id}".format(project_id=project)
    for resource, values in resources.items():
        print "  {resource}: {used}/{limit}".format(
            resource=resource, used=values["usage"], limit=values["limit"])

The results should look like this:

a1234567-a890-1234-56ae-78f90bb1c2db
  cyclades.cpu: 1/2
  cyclades.disk: 40/40
  cyclades.floating_ip: 1/1
  cyclades.network.private: 0/2
  cyclades.ram: 2147483648/2147483648
  cyclades.vm: 1/2
  pithos.diskspace: 20522192022/20522192022
a9f87654-3af2-1e09-8765-43a2df1098765
  cyclades.cpu: 4/8
  cyclades.disk: 80/120
  cyclades.floating_ip: 1/4
  cyclades.network.private: 1/5
  cyclades.ram: 4294967296/53687091200
  cyclades.vm: 3/4
  pithos.diskspace: 20522192022/53687091200

8.5.3.3. Allocate resource to a project

Example: Create an IP assigned to a specific project

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  I already know the project id
project = "a9f87654-3af2-1e09-8765-43a2df1098765"

# Initialize cyclades_network client
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
cyclades_network = CycladesNetworkClient(endpoint, TOKEN)

# Create a new floating ip
ip = cyclades_network.create_floatingip(project_id=project)
print "A new IP {0} is reserved for you".format(ip["floating_ip_address"])

Note

All “create_something” methods take an optional “project_id” argument which instructs Synnefo to register this resource to a specific project

8.5.3.4. Reassign resource to another project

Example: Reassign a storage container to different project

In the following scenario we assume that course_container is a storage container on Pithos, which is assigned to the system (personal) project and suffers from low quota limits. Fortunately, we have an extra project with enough storage available. We will reassign the container to benefit from this option.

We will check the quota limits of the project of this container and, if they are used up, we will reassign it to a different project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Our data
container_name = "course_container"
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

#  Initialize a Pithos client
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN, uuid, container_name)

#  To what project is this container assigned to?
container = pithos.get_container_info(container_name)
container_project = container["x-container-policy-project"]

#  Get quota info
quotas = astakos.get_quotas()
container_quotas = quotas[container_project]["pithos.diskspace"]
usage, limit = container_quotas["usage"], container_quotas["limit"]

if usage < limit:
    print "Quotas for container {0} are OK".format(container_name)
else:
    #  We need to reassign to another project
    new_project = "a9f87654-3af2-1e09-8765-43a2df1098765"
    pithos.reassign_container(project_id=new_project)
    print "Container {name} is reassigned to project {id}".format(
        name=container_name, id=new_project)

Note

All quotable resources can be reassigned to different projects, including composite resources (aka: depended on other cloud resources) like VMs.

8.5.4. Cyclades / Compute

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/compute-api-guide.html

A server or VM (Virtual Machine) is a complex resource: you need other resources to built it, namely CPU cores, RAM memory, Volume space and, optionally, VPNs and IPs.

8.5.4.1. List server quotas

Example: Check server-related resources and report to user

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from kamaki.clients.astakos import AstakosClient

#  Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Check quotas
total_quotas = astakos.get_quotas()
resources = (
    "cyclades.vm", "cyclades.cpu", "cyclades.ram", "cyclades.disk",
    "cyclades.network.private", "cyclades.floating_ip")
for project, quotas in total_quotas.items():
    print "Project {0}".format(project)

    for r in resources:
        usage, limit = quotas[r]["usage"], quotas[r]["limit"]
        if usage < limit:
            print "\t{0} ... OK".format(r)
        else:
            print "\t{0}: ... EXCEEDED".format(r)

8.5.4.2. Create server

Example: Create a server with an IP and a server without networking (assume the IP is reserved for use by current user)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient

#  Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Create a public server (with IP)
name = "My public server"
flavor = "420"
image = "image-id-for-debian"
IP = "123.45.67.89"
network_id_for_this_ip = 204
networks = dict(uuid=network_id_for_this_ip, fixed_ip=IP)
project = "a1234567-a890-1234-56ae-78f90bb1c2db"

public_server = compute.create_server(
    name, flavor, image, networks=networks, project=project)

#  Create a private server (without networking)
name = "My private server"
networks = []
private_server = compute.create_server(
    name, flavor, image, networks=networks, project=project)

Note

The “networks” parameter sets the connectivity attributes of the new server. If it is None (default), networking is configured according to the default policy of the cloud (e.g., automatically assign the first available public IP). To set up a server without networking: networks=[].

8.5.4.3. Wait server to built

Example: Create a new server (default settings) and wait until it is built. Then, print the server id and its IP (assume that the default networking policy of the cloud is to automatically set an IP to every new server).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Create a server (automatic IP)
name = "My public server"
flavor = "420"
image = "image-id-for-debian"
project = "a1234567-a890-1234-56ae-78f90bb1c2db"

server = compute.create_server(name, flavor, image, project=project)
print "Server status is {0}".server["status"]

new_status = compute.wait_server_until(server["id"], "ACTIVE")
if new_status != "ACTIVE":
    print "Waiting for server to build: time out..., server is in {0}".format(
        new_status)
else:
    #  Find ip
    nics = filter(lambda nic: bool(nic["ipv4"]), server["attachments"])
    ip = nics[0]["ipv4"]

    #  Create server
    print "Server {id} is now {status}\n\tName: {name}\n\tIP: {ip}".format(
        id=server["id"], status=server["status"], name=server["name"], ip=ip)

Note

The wait_server_while and wait_server_until methods work for all valid server states (ACTIVE, BUILD, STOPPED, ERROR, etc.) and can be used to block a program under some conditions. These blockers are based on the kamaki.clients.wait, which blocks for as long as a user-provided method returns true.

8.5.4.4. Query images and flavors

Example: Find the appropriate image and flavor for a server

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Find flavor with 2 cores, 20GB disk and 2048MB of ram
pick_flavor = lambda flavor: all(
    flavor["vcpus"] == 2,
    flavor["disk"] == 20,
    flavor["ram"] == 2048
)
all_flavors = compute.list_flavors(detail=True)
flavors = filter(pick_flavor, all_flavors)

#  Find images with debian in their name
pick_image = lambda image: "debian" in image["name"].lower()
all_images = compute.list_images(detail=True)
images = filter(pick_image, all_images)

#  Show results
flavor_ids = '\n\t'.join([f["id"] for f in flavors])
print "{num} flavors match\n\t{ids}".format(num=len(flavors), ids=flavor_ids)
image_ids = '\n\t'.join([i["id"] for i in images])
print "{num} images match: {ids}".format(num=len(images), ids=image_ids)

8.5.4.5. Reboot server

Example: Reboot a server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient

#  Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Get server current status
server_id = "my-server-id"
server = compute.get_server_details(server_id)
status = server["status"]

#  Shutdown server
if status == "ACTIVE":
    compute.server_reboot(server_id)
elif status == "STOPPED":
    compute.server_start(server_id)

new_status = compute.wait_sever_while(server_id, "REBOOT")
print "Server {id} is now {status}".format(id=server_id, status=new_status)

Note

Similarly you can start_server, shutdown_server as well as resize_server and delete_server.

8.5.4.6. Reassign and resize a server

Example: We need to make our server more powerful, but it’s assigned to a
project which is out of resources. We will shutdown the server, reassign it to another project and resize it to another flavor.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Data
server_id = "my-server-id"
new_flavor = "new-flavor-id-able-in-resources"
new_project = "a9f87654-3af2-1e09-8765-43a2df1098765"

#  Shutdown server
compute.shutdown_server(server_id)
compute.wait_server_until(server_id, "STOPPED")

#  Reassign and resize
compute.reassign_server(server_id, new_project)
compute.resize_server(server_id, new_flavor)

#  Start server
compute.start_server(server_id)
compute.wait_server_until(server_id, "ACTIVE")

Note

Servers must be stopped in order to resize or reassign it.

8.5.5. Cyclades / Network

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/network-api-guide.html

The Synnefo approach to the Network API diverges from the OpenStack semantics. Check the Synnefo documentation for more details.

8.5.5.1. Create private network

Example: Create a new private network between two servers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient

#  Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initialize CycladesComputeClient
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)

#  Servers
server_1 = "id-for-server-1"
server_2 = "id-for-server-2"

#  Create a new network
type_ = CycladesNetworkClient.types[0]
mynet = network.create_network(type_, name="My Network")

#  Connect servers to network
port_1 = network.create_port(mynet["id"], device_id=server_1)
port_2 = network.create_port(mynet["id"], device_id=server_2)

# Wait until ready
network.wait_port_until(port_1["id"], "ACTIVE")
network.wait_port_until(port_2["id"], "ACTIVE")

Note

In Synnefo, ports are the connections between a network and a server.

8.5.5.2. Reserve IP

Example: Check if there are free IPs, reserve one if not and use it with a server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)

#  Data
server_id = "id-for-server-1"

#  Check for unused IPs
unused_ips = filter(lambda ip: not ip["port_id"], network.list_floatingips())

#  Reserve an IP
ip = unused_ips[0] if unused_ips else network.create_floatingip()

#  Retrieve network id and IPv4 address
net = ip["floating_netowrk_id"]
fixed_ips = [dict(ip_address=ip["floating_ip_address"]), ]

#  Connect IP to server
port = network.create_port(net["id"], device_id=server_id, fixed_ips=fixed_ips)

# Wait until ready
network.wait_port_until(port["id"], "ACTIVE")

Note

IPs are connected to networks, which are connected to servers.

8.5.5.3. Create cluster

Example: Create a cluster of three servers, where only one has a public IP.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import (
    CycladesComputeClient, CycladesNetworkClient)

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)

#  Create VPN and reserve an IP
type_ = CycladesNetworkClient.types[0]
vpn = network.create_network(type_, name="Cluster Network")
unused_ips = filter(lambda ip: not ip["port_id"], network.list_floatingips())
ip = unused_ips[0] if unused_ips else network.create_floatingip()
ip_net = ip["floating_network_id"]

#  Server data
flavor = 420
image = "image-id-for-a-debian-image"

#  Create nodes
networks = [dict(uuid=vpn["id"]), ]
node_1 = compute.create_server("Node 1", flavor, image, networks=networks)
node_2 = compute.create_server("Node 2", flavor, image, networks=networks)

#  Create gateway
networks.append(dict(uuid=ip_net, fixed_ip=ip["floating_ip_address"]))
gateway = compute.create_server("Gateway", flavor, image, networks=networks)

#  Wait servers to get ready
compute.wait_server_until(node_1["id"], "ACTIVE")
compute.wait_server_until(node_2["id"], "ACTIVE")
compute.wait_server_until(gateway["id"], "ACTIVE")

8.5.6. Cyclades / BlockStorage

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/blockstorage-api-guide.html

8.5.6.1. Unplug and plug volume

Example: Create a volume for server_1, then unplug it and plug it on server_2, as you would do with a USB stick.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import (
    CycladesBlockStorageClient, CycladesComputeClient)

#  Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)

#  Servers
server_id_1, server_id_2 = "id-for-sever-1", "id-for-sever-2"

#  Initialize CycladesBlockStorageClient
service_type = CycladesBlockStorageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
blockstorage = CycladesBlockStorageClient(endpoint, TOKEN)

#  Create new volume on server_1
size_ = 20  # in GB
name = "USB stick"
usb = blockstorage.create_volume(size_, name, server_id=server_id_1)
blockstorage.wait_volume_until(usb["id"], "in_use")

#  Unplug and plug to the other server
compute.detach_volume(server_id_1, usb["id"])
blockstorage.wait_volume_while(usb["id"], "in_use")

compute.attach_volume(server_id_2, usb["id"])
blockstorage.wait_volume_until(usb["id"], "in_use")

8.5.7. Image

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/image-api-guide.html

In Synnefo, an image is loaded as a file to the storage service (Pithos+), and then is registered to the image service (Plankton). The image location is unique and can be used as an image identifier.

Image location formats:

pithos://<user_uuid>/<container>/<object path>
e.g., pithos://user-uuid/images/debian_base.diskdump

or, if the user uuid os implied
/<container>/<object path>
e.g., /images/debian_base.diskdump

8.5.7.1. Register

Example: Register the image file my-image.diskdump, currently stored locally. It will be uploaded to images.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import json

from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
from kamaki.clients.image import ImageClient

#  Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

#  Initliaze Pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
pithos.account = uuid

#  Initialize Image
service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)

#  Our data
local = "my-image.diskdump"
local_meta = "my-image.diskdump.meta"
with open(local_meta) as f:
    meta = json.load(f)

#  Upload the image and meta files
pithos.container = "images"
with open(local) as f:
    pithos.upload_object(local, f)
with open(local_meta) as f:
    pithos.upload_object(local_meta, f)

#  Register image
name, location = meta.pop("name"), meta.pop("location")
properties = meta.pop("properties")
image.register(name, location, properties=properties, params=meta)

It is a common practice to keep the image registration details in a json meta file (e.g., to register images in the future). This metafile is typically uploaded along with the image.

kamaki file cat /images/my-image.diskdump.meta
{
  "name": "Debian Base With Extras",
  "checksum": "3cb03556ec971f...e8dd6190443b560cb7",
  "updated-at": "2013-06-19 08:01:00",
  "created-at": "2013-06-19 08:00:22",
  "properties": {
    "OS": "linux",
    "USER": "root"
  },
  "location": "pithos://user-uuid/images/my-image.diskdump",
  "is-public": "False",
  "owner": "user-uuid",
  "disk-format": "diskdump",
  "size": "903471104",
  "deleted-at": "",
  "container-format": "bare"
}

8.5.7.2. List

Example: List the names and Pithos locations of the images registered by me

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.image import ImageClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)

#  Get all images owned/registered by me
images = filter(lambda img: img["owner"] == uuid, image.list_public())

print "My images:\n"
for i in images:
    print "\t{name} ({location})".format(
        name=i["name"], location=i["location"])

8.5.7.3. Unresgister

Example: Unregister and delete an image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
from kamaki.clients.image import ImageClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
pithos.account = uuid

service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)

#  Find the image by id
image_id = "my-image-id"
my_image = image.get_meta(image_id)

#  Check if it is my image
if my_image["owner"] == uuid:
    image.unregister(image_id)

    #  Delete the image files
    pithos.container = "images"
    separator = "{uuid}/{container}/".format(
        uuid=uuid, container=pithos.container)
    _, location = my_image["location"].split(separator)
    meta_object = "{0}.meta".format(location)

    pithos.del_object(location)
    pithos.del_object(meta_object)
else:
    print "This image wasn't registered by me"

Note

Unregistering an image does not delete the image dump from pithos. In order to do that, you need to have the appropriate permissions (aka, the image file must by stored on your Pithos account), so that you can delete it as a file.

8.5.8. Pithos

Synnefo API: https://www.synnefo.org/docs/synnefo/latest/object-api-guide.html Pithos+ is the storage service of Synnefo.

Each user has their own storage space, organized in containers. Each container contains objects. In most cases we can think of objects as files, but in reality they are not the same thing. Typically, it is the responsibility of the application to simulate the functionality of folders and files, if they need it.

Here is an example, where the containers are pithos, images, music and trash:

user-uuid
    pithos
        myfile.txt
        myfolder/
        myfolder/anotherfile.txt
        my-linux-distro.diskdump
    images
        debian-stable.disckdump
        my-special-image.diskdump
    music
        The Beatles - White Album/
        The Beatles - White Album/Back in the U.S.S.R.
        BoC - Music has the right to children/
        BoC - Music has the right to children/Wildlife Analysis
        BoC - Music has the right to children/An eagle in your mind
    trash
        my deleted folder/
        my deleted folder/some old file.txt
        my deleted folder/removed by accident.png

Quotas are applied at project level. Each container is registered to a project (by default, the personal/system project of the owner). Objects (“files”) inherit the project policy of the container they are in.

8.5.8.1. Initialize pithos client

Example: Initialize a pithos client to handle the objects in the container pithos

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

#  Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initliaze pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)

#  Set user UUID and Container (optional)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

pithos.account = uuid
pithos.container = "pithos"

Note

To access the objects of another user, set the account parameter to their uuid and Pithos will have access to the objects the other user allows you to see or edit.

8.5.8.2. List and information

Example Recursively list the contents of all my containers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

#  initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

#  Initliaze pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)

#  Set user UUID and Container (optional)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]

pithos.account = uuid

containers = pithos.list_containers()
for container in containers:
    pithos.container = container["name"]
    project = container["x_container_policy"]["project"]

    print "Listing contents of {container} (project: {project})".format(
        container=pithos.container, project=project)
    for object_ in pithos.list_objects():
        name = object_["name"]
        size = object_["bytes"]
        type_ = object_["content_type"]
        print "\t{name} \t{type_} \t{size} bytes".format(
            name=name, size=size, type_=type_)

The results should look like this:

Listing contents of pithos (project: a1234567-a890-1234-56ae-78f90bb1c2db)
    myfile.txt  text/plain     202 bytes
    myfolder/   application/directory   0 bytes
    myfolder/anotherfile.txt    text/plain   333 bytes
    my-linux-distro.diskdump    applcation/octet-stream    539427293 bytes
Listing contents of images (project: a9f87654-3af2-1e09-8765-43a2df1098765)
    debian-stable.disckdump     application/octet-stream    309427093 bytes
    my-special-image.diskdump   applcation/octet-stream    339427293 bytes
Listing contents of music (project: a9f87654-3af2-1e09-8765-43a2df1098765)
    The Beatles - White Album/  application/directory   0 bytes
    The Beatles - White Album/Back in the U.S.S.R.mp3  media/mpeg   3442135 bytes
    BoC - Music has the right to children/     application/directory   0 bytes
    BoC - Music has the right to children/Wildlife Analysis.mp3   media/mpeg   4442135 bytes
    BoC - Music has the right to children/An eagle in your mind.mp3    media/mpeg   23442135 bytes
Listing contents of trash (project: a1234567-a890-1234-56ae-78f90bb1c2db)
    my deleted folder/     application/directory   0 bytes
    my deleted folder/some old file.txt      text/plain   10 bytes
    my deleted folder/removed by accident.png   image/png   20 bytes

Note

In the above example, half of the projects (pithos, trash) are registered to the user’s system (personal) project, while the rest ( images, music) at another project, probably one offering more storage resources.

8.5.8.3. Upload and download

Example: Download my-linux-distro.diskdump from container “pithos” to a
local file as local.diskdump and upload it to container “images” as recovered.diskdump
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

#  Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)

user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid

#  Download from container "pithos"
pithos.container = "pithos"
source = "my-linux-distro.diskdump"
target = "local.diskdump"
with open(target, "rb+") as f:
    pithos.download_object(source, f)

#  Upload to container "images"
pithos.container = "images"
with open(target) as f:
    pithos.upload_object(source, f)

Note

The file is now stored at three locations: the container “pithos”, the container “images” and, with a different name, at the local hard disk.

Note

The upload_object and download_object methods are optimized in many ways: they feature dynamic simultaneous connections and automatic resume. In this case for instance, the data of the uploaded file will not be uploaded, as they are already on the server.

8.5.8.4. Move / Copy / Delete

Example: Move /pithos/my-linux-distro.diskdump to trash, delete
/images/my-pithos-distro.diskdump and then copy it from trash.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)

user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid

#  Move to "trash"
object_ = "my-linux-distro.diskdump"
pithos.move_object("pithos", object_, "trash")

#  Delete from "images"
pithos.container = "images"
pithos.del_object(object_)

#  Copy from "trash" to "images"
new_object = "recovered.diskdump"
pithos.copy_object("trash", object_, "images", new_object)

This is the status after the execution of the script above:

user-uuid
    pithos
        myfile.txt
        myfolder/
        myfolder/anotherfile.txt
    images
        debian-stable.disckdump
        my-special-image.diskdump
        recovered.diskdump
    music
        The Beatles - White Album/
        The Beatles - White Album/Back in the U.S.S.R.
        BoC - Music has the right to children/
        BoC - Music has the right to children/Wildlife Analysis
        BoC - Music has the right to children/An eagle in your mind
    trash
        my deleted folder/
        my deleted folder/some old file.txt
        my deleted folder/removed by accident.png
        my-linux-distro.diskdump

8.5.8.5. Reassign container

Example: Reassign container “images” to the same project as the one of “pithos”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient

AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)

service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)

user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid

#  Get the project containers we care for
containers = filter(
    lambda c: c["name"] in ("pithos", "images"),
    pithos.list_containers())

#  Construct dict of the form {CONTAINER_NAME: PROJECT_ID, ...}
projects = dict([(
    c["name"],
    c["x_container_policy"]["project"]) for c in containers])

#  Check projects and reassign if needed
if projects["pithos"] != projects["images"]:
    pithos.container = "images"
    pithos.reassign_container(projects["pithos"])