Table database

Table databases in SecreC

Table database

SecreC supports a basic file-based table database system intended for storing and organizing large amounts of data to be privately processed in SecreC.

Before creating a table a connection must be opened to the data source with the function tdbOpenConnection. The name of the data source is set in Sharemind's configuration. Before opening a new connection to a different data source make sure the old connection has been closed with tdbCloseConnection. A simple table with a uniform data type can be created with the function tdbTableCreate.

Listing 1: Creating an empty table

//creating an empty table with 3 columns of type int32 and protection domain shared3p
 
import shared3p;
import shared3p_table_database;
import stdlib;
import table_database;
 
domain pd_shared3p shared3p;
 
void main () {
        string datasource = "DS1";
        string table_name = "SimpleTable";
 
        tdbOpenConnection (datasource);
        pd_shared3p int32 data_type;
 
        tdbTableCreate (datasource, table_name, data_type, (uint64) 3);
}

Data is added to the table one row at a time with the function tdbInsertRow. Each row is a vector with every element corresponding to a column in the table. Data is read from the table one column at a time with the function tdbReadColumn. The identifier of the column can be the column's index or the name of the column.

Vector Map

A vector map (also referred to as a value map or a vmap) is a data structure for making more complicated tables with column identifiers and multiple data types. A vmap can contain four different types of information: types, strings, values and indexes. A vmap is similar to a Python dictionary as the vmap contains parameters and data associated to those parameters.

Values added to the vmap can be either fixed lenght or variable length. Variable length means that the size of values must not be uniform in one column e.g. strings or vectors with variable length. Values are stored in the vmap as batches. Each batch covers all columns but one row. Data can be added one entry at a time with tdbVmapAdd{Type/String/Value/Index} for fixed length data or with tdbVmapAddVlen{Type/Value} for variable length data.

Creating a table from a vector map

When creating a table instead of specifing a data type and the number of columns a vmap can be used. The vmap must contain a type and string for every column with the parameters "types" and "names" respectively. Data can be inserted to the table with tdbInsertRow but instead of a vector a vmap can be used. The vmap must have values with the parameter "values" that are the same type as their respective column in the table. Every batch in the vmap corresponds to a single row in the table.

Listing 2: Creating a table with a vector map

import shared3p;
import shared3p_table_database;
import stdlib;
import table_database;
 
domain pd_shared3p shared3p;
 
void main() {
        string ds = "DS1"; // Data source name
        string tbl = "table"; // Table name
 
        // Open database before running operations on it
        tdbOpenConnection(ds);
 
 
        // Check if a table exists
        if (tdbTableExists(ds, tbl)) {
                // Delete existing table
                tdbTableDelete(ds, tbl);
        }
 
        // We want to create a simple table with three columns:
        //
        //       -----------------------------------------------------------
        // Name: |       "index" |      "measurement" | "have_measurement" |
        // Type: | public uint64 | pd_shared3p uint64 |   pd_shared3p bool |
        //       -----------------------------------------------------------
        //       |             0 |                  0 |               true |
        //       |             1 |                 10 |               true |
        //       |             2 |                 20 |               true |
        //       |             3 |                 30 |               true |
        //       |             4 |                 40 |               true |
        //       -----------------------------------------------------------
 
        // Create a new "vector map/value map" for storing arguments to the table
        // creation call.
        uint params = tdbVmapNew();
 
        // Column 0, name "index", type public uint64
        {
                uint64 vtype;
                tdbVmapAddType(params, "types", vtype);
                tdbVmapAddString(params, "names", "index");
        }
 
        // Column 1, name "measurement", type pd_shared3p uint64
        {
                pd_shared3p uint64 vtype;
                tdbVmapAddType(params, "types", vtype);
                tdbVmapAddString(params, "names", "measurement");
        }
 
        // Column 2, name "have_measurement", type pd_shared3p bool
        {
                pd_shared3p bool vtype;
                tdbVmapAddType(params, "types", vtype);
                tdbVmapAddString(params, "names", "have_measurement");
        }
 
        // Create the table
        tdbTableCreate(ds, tbl, params);
 
        // Free the parameter map
        tdbVmapDelete(params);
 
        // Insert some data
        uint nrows = 5;
        params = tdbVmapNew();
 
        for (uint i = 0; i < nrows; ++i) {
                uint64 index = i;
                pd_shared3p uint64 measurement = i * 10;
                pd_shared3p bool have_measurement = true;
 
                if (i != 0) {
                        // This has to be called in-between rows
                        tdbVmapAddBatch(params);
                }
 
                tdbVmapAddValue(params, "values", index);
                tdbVmapAddValue(params, "values", measurement);
                tdbVmapAddValue(params, "values", have_measurement);
        }
 
        tdbInsertRow(ds, tbl, params);
 
}

Erasing tables

Before creating a new table it is recommended to check if a table with the same name already exists and delete it if it does. A table can be deleted with tdbTableDelete. After the program finishes SecreC automatically deletes all existing vector maps but for safety concerns it is advised to do this manually.