com.webos.service.db

API Summary

Enables apps to store persistent data.

DB8 is a user space service that provides access to the webOS database. Access to the database APIs is provided over the luna-service bus.

Formally, a database refers to a set of related objects and the way it is structured or organized. Access to these objects is usually provided by a luna API that allows applications to interact with the database and one with another. It also provides access to the objects in the database (although there may be restrictions that grant or limit access to particular data). DB8 provides various functions that allow to enter, store, and retrieve large volumes of information as well as provides the interface to manage how that information is organized.

Designing a database:

The first task of a database designer is to produce a conceptual model that reflects the structure of data as it is going to be stored in a database. A common approach to this is to develop a Kind that implements the relevant data structures within the database.

Example of Kind creation:

In the example above, we create a Kind with an ID: "com.webos.service.test:1" (this is similar to a schema name).

The owner of this Kind service is com.webos.service.test. The owner of a Kind has permissions to calculate quotas, grant permissions to modify or delete a Kind, grant permissions to other services for Kind data, etc.

We have also specified two indexes for the Kind. An index is a data structure that allows data retrieval operations on a database at the cost of additional writes and storage space to maintain the index data structure. Indexes are used to quickly locate data without having to search every row in a database every time a database is accessed. Indexes are created using one or more columns of a database object, providing the basis for both rapid random searches and efficient access of ordered records.

The number after the colon in the ID is the Kind Revision number. When a service updates a Kind, the service should increment its revision number to indicate that the schema has changed.

Philosopher's stone (objects):

DB8 operates with objects. Objects have fields. Every object that is stored within DB8 always has an ID and a revision.

An object ID uniquely identifies an object within a database. The object ID does not repeat within a database.

Object revisions:

Object revision is a number, that determines "version of an object in the database". This number is required to avoid collisions when two or more services simultaneously try to update an object at the same time.

For all new objects, the database will append a _rev field, which indicates the object revision. By default, the database will set object revision _rev field to Database Revision (Number, that atomically increments on any database modification).

If an object already exists in the database, the database engine will automatically increment the object revision number on each successful put/merge API call for this object.

Explanation by example: com.webos.service.test and com.webos.service.othertest applications have access for read/write to Kind com.webos.service.test:1. Both applications, at the same time, read an object from com.webos.service.test:1 kind object with ID ++JZk1sXYW7lCdqU and revision 777.

Both applications execute a merge command like the following:

Following is the merge call response, that will be received by one of the applications - (may be com.webos.service.test or com.webos.service.othertest, depends who is more lucky):

The other application will get the following error:

This error indicates that someone else has already updated this object. After receiving this error, the application MUST execute the get method to get the object again with a new _rev, and then try to re-execute the merge query.

Create:

To store an object in the database, a client can use the put method.

Complexity guaranteed for a put API call is log(n). 

Update (replace):

The put method updates an object. Update by default will replace the database object with the new one (in other words, if a new object does not have some fields that were present in the old database object, those old database fields will be removed)

As an example, consider the following object in a database:

And the following luna call is made:

If the call is successful, the database will contain:

Update (Merge):

Sometimes a service does not need to replace an object in the database, but simply needs to add a field to a database object. For this type of update, a client can call the merge method instead of the put method.

For example, if we have an object:

And the following luna call is made:

If the call is successful, the database will contain:

Trick with put/merge and object revision:

The _rev object field shows the revision number of an object. But what happens if an application wants to track updates to specific field/fields? DB8 can do that.

The application should modify the Kind and add revSets.

Revision set (revSets) is a set of object field names, that the database should track for changes. If the tracked field(s) are modified, the database will increment the number.

The above modification to the Kind tells DB8 to do the following:

1. Add the field1_rev field to each object
2. Increase its value when the field1 object is modified.

The default value of field1_rev will be the same as object _rev.

Example:

Lets create a new object:

In the database, the object will be stored as:

The database then adds the field1_rev field to the object.

When an application updates field1, DB8 will automatically increment the field1_rev field:

The database will contain:

Now, if we update the field2 only, the database will not increment the field1_rev field, as it does not track updates to field2.

The database will contain:

Queries:

DB8 guarantees complexity for all CRUD (create, read, update, delete) API calls.

To retrieve data, a client can call either the get or the find method.

• The get method provides the ability to get an object by the object id.
  Complexity guaranteed for get API call is log(n)
• The find method provides the ability to get objects by condition. The find query must use the index fields of an object for searching.
  Complexity guaranteed for the find API call is log(n)
• The search method provides the ability to get objects by condition. The search Query uses any object field for searching.
  Complexity guaranteed for the search API call is n.

It may not be clear, why the find method does not allow operations, like filter or sub-string search. Think about it this way: DB8 always guarantees complexity for all of its API calls without exception. An operation like the filter can search by object fields that do not have indexes. In other words, DB8 would have to iterate over n objects to process the resulting set. However, since DB8 guarantees a complexity of log(n) to the find call, it does not allow the filtering operation.

Some quick examples for find (with collate and limit):

Quick example for search:

Pages:

Processing of result sets that contain a lot of objects (thousands, millions, etc) can consume all available memory. Assume that a database has 1 million objects and we need to build a JSON representation of those objects on a target system that has only 1M of free RAM. How can that be achieved? To handle such a scenario, DB8 will output result sets by pages. Each page will have a limit on the maximum number of objects it can contain. If an application does not specify a limit in find/search, DB8 will add a default limit. On most platforms, the default limit value is 500 objects per page, but this can be reconfigured.

For example, we retrieve data from a DB8 database on a Tamagochi device (each page will contain only two objects):

DB8 response:

The next parameter in the response indicates the ID of the next page in the result data set (internally, DB8 interprets this id as a very tricky offset). To get the next object in the result data set, execute query with a page parameter:

And DB8 will response with next 2 objects and id of the next page:

If DB8 does not have any objects in the result set and the last retrieved page contains the final objects from the result set, DB8 will not provide the next parameter in the response.

Complex Queries:

For this section we are using the following test Kind:

Example database set for this documentation section:

For test queries, we use these put API calls:

In the Queries section you can find how to create an index for one field and execute some trivial queries, like:

If an application developer tries to search using two fields at the same time:

DB8 will return "Sorry, I don't know such index". This error is returned because the DB8 Planner is not very intelligent and will never try to guess the index.

An application developer should provide complex indexes for such complex quires. To get an object using 2 fields (third object from example set):

The client should provide the Kind for such index like following:

(See at top of this documentation section for full luna comand for kind creation.)

After modification of the Kind, an application can execute a query like this:

Complex Queries Tricks:

Imagine, that an application developer extends a Kind with a complex index like the one below, and then tries to execute it:

DB8 will return "Index not found", even if you have created a complex index for field1 and field2. In the example below, the index has the name field12.

To work around this, you should also create a reverse index field21:

After such a tricky index creation, DB8 can successfully search for data.

Watch:

Sometimes a client may want to be notified about an object in a database that satisfies some condition. For this, the client can use the watch method.

Example:

If no object exists with the field processed=false for Kind com.webos.service.test:1, the client will be blocked in the watch method. The watch will be unblocked, when an object that satisfies a query will appear in a database. In the example above, the database will have object for com.webos.service.test:1 with field processed=false (if someone will insert object or modify field for existing object).

Note about blocking: For a blocking watch API call, the caller cannot do anything until the watch API call returns. It is very similar to system calls blocks. 

Example of watch response, when it is unblocked:

The fired param in the response informs the client that DB8 contains one or more objects that satisfy the watch query and that the client can retrieve those objects.

Sample workflow for the watch method:

Assume that you are developing a chat application.

• The chat application creates a DB8 Kind com.chat.messages:1
• The chat application will have 2 independent threads: A UI thread that shows messages and a reader thread, that receives messages from the network.

1. Phase: Application startup
   UI thread reads all objects from com.chat.messages:1 Kind and shows the messages to the end user.
2. Phase: UI Blocks till new message arrive
   UI thread calls DB8 watch method with query: processed=false. If the database does not have such objects, the UI thread is blocked in the watch API call.
3. Phase: reader thread receives a new message
   The reader thread receives messages over the network, stores those messages into com.chat.messages:1 with set field processed=false.
4. Phase: UI thread unblocked
   The Database unblocks watch API call (UI thread was blocked in watch API call on step 2), the UI executes the query find with parameter processed=false
5. Phase: UI thread processes the new message
   For each new object in the result set, the UI thread shows those object to the end user. To indicate, that each new object was processed and shown to the end user by the UI, UI thread set field processed=false and merge each object into database.
6. Phase: UI thread is blocked again
   UI go to step 2.

Collation:

Information is displayed in sorted order to enable users to easily find the items they are looking for. However, users of different languages might have very different expectations of what a "sorted" list should look like. Not only does the alphabetical order vary from one language to another, but it also can vary from document to document within the same language. For example, phone book ordering might be different than dictionary ordering. String comparison is one of the basic functions most applications require, and yet implementations often do not match local conventions. Collation provides string comparison capability with support for appropriate sort orderings for each of the locales needed by a client.

Collation rule:

A RuleBasedCollator is built from a rule string that changes the sort order of some characters and strings relative to the default order. An empty string (or one with only white space and comments) results in a collator that behaves like the root collator.

Customization is specified via a string containing a set of rules. ICU implements the (CLDR) For more details see LDML collation rule syntax. 

Each rule contains a string of ordered characters that starts with an anchor point or a reset value. For example, "&a < g", places "g" after "a" and before "b", and the "a" does not change place. This rule has the following sorting consequences:

Note: Only the word that starts with "g" has changed place. All the words sorted after "a" and "A" are sorted after "g".

This is a non-complex example of a custom rule. Custom rules consist of zero or more rules and zero or more options. There must be at least one rule or at least one option. The rule syntax is discussed in more detail in the following sections.

Note: The custom rules override the UCA ordering. In addition, if a character is reordered, it automatically reorders any other equivalent characters. For example, if the rule "&e<a" is used to reorder "a" in the list, "á" is also greater than "é".

Syntax:

The following table summarizes the basic syntax necessary for most usages:

For more information, see:

• http://userguide.icu-project.org/collation
• http://userguide.icu-project.org/collation/customization

Collation can be specified in the putKind or the find method.

Example of using collator with putKind - set default collation for kind:

Internally, DB8 process strings via ICU. All ICU documentation about strings somewhat applies to DB8 string processing.

Permissions:

DB8 is the central storage for applications. Every application can not only store/retrieve data by itself, but it can also share and interact with other applications.

Application can share kind and grant permissions for any CRUD operator. For more information, see putPermissions method.

Quotas:

The function of quotas is to allocate limited disk space in a reasonable way for target device. Those limits are set by product developers via DB8 configuration files.

In other words, quotas limit the amount of disk space that can be used by a certain application.
Every process can see its quota usage by using the quotaStats method.

Encoding:

When the DB8 daemon starts, it uses UTF8 en_US.en_US encoding which is its default encoding.

DB8 subscribes for com.webos.service.systemservice/getPreferences '{ "keys" : "locale"}' . If the systemservice does not exist on the platform or returns an error, DB8 will continue use UTF8 en_US.en_US encoding by default.

Disk Storage Logic:

When DB8 starts, it creates 2 activities in Activity Manager:

• Scheduled space check
• Garbage collector (purge)

Activity for space check will check for available storage space. If the free storage space is too low, DB8 will reject all new CRUD API calls with quota exceeded error. If the storage has enough free space, then on the next scheduled space check DB8 will continue to process CRUD API calls.

Garbage collector:

By default, DB8 does not remove objects from its database, it only marks it for future delete. DB8 use this logic, as an update operation takes less resources compared with delete. Physical objects are removed when processed by the DB8 garbage collector.

The garbage collector can be called in 2 ways: by ActivityManager or directly by service.

On startup, DB8 dynamically registers an activity to call the garbage collector.

If the client wants to delete an object and immediately free storage space, the client can specify the purge option in the del method. The del API call with the purge parameter can be potentially slow.

Complexity for garbage collector: m*log(n), where m - count of objects marked for removal.

Error handling:

If DB8 cannot process method calls (internal error, bad params, no resources, etc) it always returns error in the following format:

• returnValue - If an error occurs, it  will contain false
• errorCode - Code that indicates the cause of  the error
• errorText   -  Text representation of the error code

Error codes and descriptions are provided for each method call. For example, if the client calls the load method with a bad parameter:

The above response indicates, that:

• returnValue: false - method call failed
• errorCode: 2 - See the load method for more details on error code 2
• errorText: No such file or directory - A quick textual explanation of the error

Basic API functions:

It provides the following data management functions:

• Add objects
• Update objects
• Delete objects
• Query objects 
• Manage access to objects within the database

batch

Description

The compact method invokes the low level garbage collector. When DB8 executes a del operation without Purge:True param, it only marks the object as to be deleted. The object will be removed by DB8, when the garbage collector is called. To call the garbage collector manually, the client can call the compact method.

This command is implemented only for the LevelDB and Sandwich engine.

Parameters

None

Call Returns

Error Codes Reference

Example

Description

The del method deletes JSON data objects from the database.

Apps can specify the objects to be deleted by providing:

• a set of IDs to be deleted
• a DB8 query

Parameters

Call Returns

Error Codes Reference

Example

Description

The delKind method deletes a Kind from the database. Deleting a Kind deletes ALL data objects of that Kind.

Parameters

Call Returns

Error Codes Reference

Example

Description

The dump method is used to backup a database. The file created by the dump method typically contains JSON statements to recreate all of the Kinds and data of the database from which they have been dumped.

Parameters

Call Returns

Error Codes Reference

Example

Description

The find method returns a set of objects that match the query specified in the query parameter.

The app can specify the number of results to return. However, if the app does not want to specify a limit, it can set the count parameter to true. This will cause the find method to return the total number of results.

The app can also request to be notified if any of the returned results from the query change in the future. In order to receive change notifications, set the watch parameter to true.

The find method supports distinct groups enabling the app to remove duplicate objects.

Parameters

Call Returns

Error Codes Reference

Example

Description

The ​get method retrieves JSON data objects by ids. This is the fastest way to retrieve data.

Parameters

Call Returns

Error Codes Reference

Example

Description

Get profiling data for applications. Profile data includes the queries made and related information such as request time and response time.

Parameters

Call Returns

Error Codes Reference

Example

Description

The load method restores a database from a dumped JSON file.

Parameters

Call Returns

Error Codes Reference

Example

Description

The merge method updates the properties of existing objects.

The objects can be specified in one of the following ways:

• A query
• An array of IDs

Parameters

Call Returns

Error Codes Reference

Example

Description

The mergePut method updates the properties of existing objects. If an object doesn't exist, a new one will be created in the database.

The object to be updated can be specified in one of the following ways:

• A query
• An array of IDs

Parameters

Call Returns

Error Codes Reference

Example

Description

Enables or disables DB8 profiling for an application. When enabled, profiling data is stored for the application.

You can enable DB8 profiling using one of the following approaches:

- Self enabling: In this approach, the 3rd-party application enables profiling on itself (see example 1).

- Enable profiling using admin-privileged applications: In the approach, an admin-privileged application controls DB8 profiling for 3rd-party applications. Use this approach when you want the admin to control the profiling of 3rd-party applications. You can choose to enable profiling for a single application or for all 3rd-party applications. See example 2 and 3.

Parameters

Call Returns

Error Codes Reference

Example

Description

The purge method invokes the garbage collector. The purge method will:

• Remove all objects that were marked for deletion
• Perform a space check, and remove all temporary data

Parameters

Call Returns

Example

Description

The purgeStatus method returns the status of the last run purge command. If the last run purge command was successful, the objects were permanently deleted, and the purgeStatus method will return the updated database revision number.

Parameters

None

Call Returns

Example

Description

The put method stores JSON data objects of a particular Kind into the database. The put method will:

• Assign an id field to each object, if it was not set.
• Return the id and rev for each stored object.

Parameters

Call Returns

Error Codes Reference

Example

Description

The putKind method registers a kind with the database.

Kinds define the owner, and the indexes for a JSON data object. Indexes can be composed of single or multiple properties. When you create your index, be aware that queries can only return results that are indexed, and are contiguously ordered.

If your app or service wants to be notified only when a subset of an object's properties are updated, then you can use revision sets.

If your app or service creates objects that other apps or services need to access, then see the putPermissions method for more information.

Parameters

Call Returns

Error Codes Reference

Example

Description

The putPermissions method enables other apps or services to access an app's stored DB8 data. The app can give permissions to access data objects of a specific Kind.

Parameters

Call Returns

Error Codes Reference

Example

Description

The putQuotas method provides the ability to update a quota's current service limits at runtime. This service is used by private webOS services to increase/decrease quotas for critical webOS services.

Parameters

Call Returns

Example

Description

The quotaStats method returns information about a service's used limits.

Parameters

None

Call Returns

Example

Description

The removeAppData method removes all data associated with the given owner. In other words, the method removes all Kinds that have specified owner.

Parameters

Call Returns

Error Codes Reference

Example

Description

When a client service creates objects that have references between each other, the service can ask the database through the reserveIds method to regenerate ids of objects. The client service can use such ids as objects, and DB8 will use those ids when objects are inserted into the database.

By default, DB8 configured to reserve maximum [0:1000] ids, but this limit can vary depending on the platform.

Parameters

Call Returns

Error Codes Reference

Example

Description

Unlike the find method, the search method supports the "?" (question mark) operator. The ? operator can be used for full-text searching. However, the search method is significantly slower, and should only be used for full-text type-down search. The search method should not be used for retrieving results that are going to be scrolled in a list.

The search method supports:

• Ordering by any property.
• Distinct method to remove duplicate objects.
• %% operator in filter to search in a sub-string.

The search method has some limitations:

• ​There must be an index for the field you are searching on.
• The search operation looks for words beginning with the search string.

Parameters

Call Returns

Subscription Returns

Error Codes Reference

Example

Description

The stats method returns detailed information about the storage space used by every service.

Parameters

Call Returns

Example

Description

The watch method watches for updates to the database that would change the results of a query.

Parameters

Call Returns

Error Codes Reference

Example

BatchOperation

Method and params for batch operation.

BatchResponse

Response to batch operation.

Query

Defines a db8 query.

IndexClause

Used in the putKind call for creating kind object indexes. Note that indexes determine the type of queries your app can make. See Queries and Indexing for more information. Set the incDel flag to true if you want future queries to return marked as deleted objects. Objects are not completely deleted until an administrative purge operation takes place.

RevSetClause

Defines a revision set - subset of an object's properties that your app can be notified about when one of the properties is modified.

operation

Object used to represent a batch operation to run.

Note: This object has different format as target. Following object example is for the target.

putQuotas

Represent Kind owner and maximum allowed DB8 storage usage.

quotaStatsResult

Information about used quotas.

statsKindResult

Information about kind usage.

WhereClause

Defines a SQL-like JSON where clause for a db8 Query.

FilterClause

Definition of the Filter clause that is part of the Query object.

Results

Contains "id" and "rev" fields for the JSON data object.

IndexPropClause

Defines index property for IndexClause

RevSetPropClause

A property in a RevSetClause.