com.webos.service.ai.voice

API Summary

This service is used to recognize voice input.

It can use and customize the configurations defined by the configuration API (com.webos.service.config).

Overview of the API

Note
Please refer to Google Assistant Setup Guide for more detail.

In webOS OSE, before using the ai.voice service, you need to set up Google Assistant on your device. The steps are detailed here:

Step A: Create credentials.json file.

Create a Google account.
Go to the Actions Console and perform the following steps:
1. Create a new project.
2. Register your device model and download the OAuth-2.0 credentials file. (See instructions.)
  Note: It is recommended that you select the device type as Auto to avoid errors during the setup.
After downloading the file,
1. Move it from your local PC to /etc/googleAssistant/client_secret.json on your device using the command:
  $ scp <downloaded json file> root@<your device ip>:/etc/googleAssistant/client_secret.json
2. From the device, run the following script:
  $ cd /etc/googleAssistant
  $ ./get_credentials.sh
Go to the URL generated by the above script. It provides some code.
Copy the code and paste it on the device terminal and run the code. It creates the 'credentials.json' file.
Enable Google Assistant API, and configure the consent screen and activity controls for your project/account. (See instructions.)

Step B: Register the device ID.

Open device_id.json file.
$ cd /etc/googleAssistant
$ vi ./device_id.json
Modify the fields:
- id - Input the device instance string. Typically, 'my_webos'.
- model_id - Input the device model ID registered in Step A.
Register the device using the command:
$ ./register_device_id.sh

Step C: Input the device model and device ID for AI service.

Enter the command:
$ vi /etc/systemd/system.conf.d/ai.conf
Modify the fields:
- GOOGLEAI_DEVICE_MODEL: Input the device model ID registered in Step A
- GOOGLEAI_DEVICE_ID: Input the device instance string registered in Step B

Step D: Reboot device or restart service daemon.

Enter the command:
$ reboot
OR,
$ systemctl restart ai

Step E: Launch the service using the start() method.

Note: Google assistant also supports custom device actions.

For more information on the custom actions file, refer to /etc/googleAssistant/action.en.json.
Requires the gactions CLI tool.

Methods

start

ACG: ai.management

Description

Starts the service based on the mode selected.

Note: This method is only for allowed clients.

Parameters

Name	Required	Type	Description
mode	Required	String	Select the service mode. Possible values are: continuous - The service will restart automatically when a conversation ends. single - The service will be stopped when a conversation ends.
keywordDetect	Required	Boolean	Indicates if a keyword is to be used for activating speech recognition. Possible values are: true: Activated by a keyword. The default keyword is "Snowboy". To change the keyword model, refer snowboy.kitt.ai. Download custom the keyword model and move it to /etc/googleAssistant/ on your device. Then, modify KEYWORD_DETECT_MODEL in /var/systemd/system/env/ai.env. false: Activated by a button press.

Name

Required

Type

Description

mode

Required

String

Select the service mode. Possible values are:

continuous - The service will restart automatically when a conversation ends.
single - The service will be stopped when a conversation ends.

keywordDetect

Required

Boolean

Indicates if a keyword is to be used for activating speech recognition. Possible values are:

true: Activated by a keyword. The default keyword is "Snowboy".
To change the keyword model, refer snowboy.kitt.ai.
Download custom the keyword model and move it to /etc/googleAssistant/ on your device. Then, modify KEYWORD_DETECT_MODEL in /var/systemd/system/env/ai.env.
false: Activated by a button press.

Call Returns

Name	Required	Type	Description
returnValue	Required	Boolean	Indicates the status of operation. Possible values are: true - Indicates that the operation was successful. false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.
errorCode	Optional	Number (int8_t)	The error code for the failed operation.
errorText	Optional	String	Indicates the reason for the failure of the operation. See the "API Error Codes Reference" section for details.

Name

Required

Type

Description

returnValue

Required

Boolean

Indicates the status of operation. Possible values are:

true - Indicates that the operation was successful.
false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.

errorCode

Optional

Number (int8_t)

The error code for the failed operation.

errorText

Optional

String

Indicates the reason for the failure of the operation. See the "API Error Codes Reference" section for details.

Example

Example scenario

# luna-send -n 1 -f luna://com.webos.service.ai.voice/start '{"mode": "continuous", "keywordDetect": true}'

Response:

{
"returnValue":true
}

getState

ACG: ai.voice

Description

Gets the current state of the service. It can have the following states:

idle: Before start.
waitActivation: Wait for keyword activation.
recording: Uploading audio to the server for speech recognition.
thinking: Waiting for results from server.
answering: Answering to the request through the speaker.

com.webos.service.ai.voice state diagram

Parameters

Name	Required	Type	Description
subscribe	Optional	Boolean	Subscribe for notifications. Possible values are: true - Get notifications. false - Notifications are not required. Default value: false

Name

Required

Type

Description

Optional

Boolean

Subscribe for notifications. Possible values are:

true - Get notifications.
false - Notifications are not required.

Default value: false

Call Returns

Name	Required	Type	Description
returnValue	Required	String	Indicates the status of operation. Possible values are: true - Indicates that the operation was successful. false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.
subscribed	Optional	Boolean	Indicates if subscribed to get notified. Possible values are: true - Get notifications false - Notifications are not required
state	Required	String	Indicates the running state of the service. Possible values are: idle waitActivation recording thinking answering
errorCode	Optional	Number (int8_t)	The error code for the failed operation.
errorText	Optional	String	Indicates the reason for the failure of the operation. See the "API Error Codes Reference" section for details.

Example

Example scenario

# luna-send -n 1 -f luna://com.webos.service.ai.voice/getState '{}'

Response:

{
   "returnValue":true,
   "state":"idle"
}

# luna-send -i -f luna://com.webos.service.ai.voice/getState '{"subscribe": true}'

Response:

{
   "returnValue":true,
   "state":"idle"
}
{
   "state":"waitActivation"
}
{
   "state":"recording"
}
...

getResponse

ACG: ai.voice

Description

Returns speech recognition results. This method is asynchronous.

Parameters

Name	Required	Type	Description
subscribe	Required	Boolean	Subscribe for notifications. Possible values are: true - Get notifications. false - Notifications are not required. Default value: false

Name

Required

Type

Description

Required

Boolean

Subscribe for notifications. Possible values are:

true - Get notifications.
false - Notifications are not required.

Default value: false

Call Returns

Name	Required	Type	Description
returnValue	Required	Boolean	Indicates the status of operation. Possible values are: true - Indicates that the operation was successful. false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.
subscribed	Optional	Boolean	Indicates if subscribed to get notified. Possible values are: true - Get notifications. false - Notifications are not required.

Name

Required

Type

Description

returnValue

Required

Boolean

Indicates the status of operation. Possible values are:

true - Indicates that the operation was successful.
false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.

subscribed

Optional

Boolean

Indicates if subscribed to get notified. Possible values are:

true - Get notifications.
false - Notifications are not required.

Subscription Returns

Name	Required	Type	Description
provider	Optional	String	Specifies the AI service provider that returns the results of speech recognition. In this case, it will always return "googleassistant".
response	Optional	String	The results of speech recognition. Different providers will provide results in different format.

Name

Required

Type

Description

provider

Optional

String

Specifies the AI service provider that returns the results of speech recognition.

In this case, it will always return "googleassistant".

response

Optional

String

The results of speech recognition.

Different providers will provide results in different format.

Example

Example scenario

Sample for webOS OSE users (when using googleassistant)

# luna-send -i -f luna://com.webos.service.ai.voice/getResponse '{"subscribe": true}'

Response:

{
"returnValue":true
}

Googleassistant for webOS OSE case:

Example subscribe for Trigger Word Detection:

{
   "provider":"googleassistant",
   "response":{
      "keywordDetected":1
   }
}

Example subscribe for Automatic Speech Recognition:

{
   "provider":"googleassistant",
   "response":{
      "partial":"what"
   }
}
{
   "provider":"googleassistant",
   "response":{
      "partial":"what's"
   }
}
{
   "provider":"googleassistant",
   "response":{
      "partial":"what's the"
   }
}
{
   "provider":"googleassistant",
   "response":{
      "partial":"what's the weather"
   }
}
{
   "provider":"googleassistant",
   "response":{
      "displayText":"It looks pleasant outside Currently in Seoul it's 79 and partly cloudy. The forecast today is 78 and mostly cloudy. ( More on weather.com )"
   }
}

stop

ACG: ai.management

Description

Stops the service. Make sure to call this method before starting another engine.

Note: This method is only for allowed clients.

Parameters

None

Call Returns

Name	Required	Type	Description
returnValue	Required	Boolean	Indicates the status of operation. Possible values are: true - Indicates that the operation was successful. false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.
errorCode	Optional	Number (int8_t)	The error code for the failed operation.
errorText	Optional	String	Indicates the reason for the failure of the operation. See the "API Error Codes Reference" section for details.

Name

Required

Type

Description

returnValue

Required

Boolean

Indicates the status of operation. Possible values are:

true - Indicates that the operation was successful.
false - Indicates that the operation failed. Check the "errorCode" and "errorText" fields for details.

errorCode

Optional

Number (int8_t)

The error code for the failed operation.

errorText

Optional

String

Indicates the reason for the failure of the operation. See the "API Error Codes Reference" section for details.

Example

Example scenario

# luna-send -n 1 -f luna://com.webos.service.ai.voice/stop '{}'

Response:

{
"returnValue":true
}

API Error Codes Reference

Error Code	Error Text	Error Description
0	Not error	Not error.
-1	Unknown error	Unknown error.
-2	Invalid parameters	This error occurs when api parameters are incorrectly.
-3	Invalid configurations	This error occurs when system missed the environment files.
-5	Network error has occurred	Network error has occurred.
-6	Not supported	Not supported api.
-7	Service had not started	Service had not started.
-8	Service had already started	Service had already started.
-9	Microphone error has occurred	Microphone error has occurred.
-10	Audio error has occurred	Audio error has occurred.
-11	Camera error has occurred	Camera error has occurred.