Install and run
The Ably Database Connector (ADBC) sends updates from your database to frontend clients through Ably channels using the outbox pattern. When you update data in your database, the changes are recorded in an outbox table. The Database Connector detects these changes and publishes them as messages to specified channels. Client applications using the Models SDK subscribe to these channels to receive updates to their local data.
The Database Connector is designed to run with your existing database. However, you can run a new database if you prefer.
Pull the Database Connector
The Database Connector is a Docker container image, making it compatible with any cloud infrastructure platform that can run Docker containers.
The following is an example of how to pull the Database Connector image:
docker pull ghcr.io/ably-labs/adbc:latest
CopyCopied!
Run an existing database
Use Docker Compose to run an existing PostgreSQL instance with the Database Connector. By default, Docker Compose automatically generates a PostgreSQL instance for you.
Docker Compose sets up and runs a local development environment. You can create and start your PostgreSQL database and an instance of the Database Connector on your local machine.
Setup in Docker Compose
To begin, create a docker-compose.yml
file with the following contents:
version: '3'
services:
adbc:
image: ghcr.io/ably-labs/adbc:latest
env_file:
- adbc.env # load config from env file
# Uncomment below if you want to load config from your adbc.yaml file,
# which takes precedence over config from the env.
# volumes:
# - ./adbc.yaml:/adbc.yaml:ro # mount yaml config file
depends_on:
postgres:
condition: service_healthy
networks:
- adbc_network
postgres:
image: postgres:11-alpine
ports:
- 5432:5432
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
healthcheck:
test: ["CMD", "pg_isready", "-q", "-d", "postgres", "-U", "postgres"]
interval: 2s
retries: 30
networks:
- adbc_network
volumes:
- adbc_postgres_data:/var/lib/postgresql/data
volumes:
adbc_postgres_data:
networks:
adbc_network:
CopyCopied!
Configure in Docker Compose
Create an adbc.env
file with, at a minimum, the following configuration options:
ADBC_ABLY_API_KEY=<loading API key, please wait>
ADBC_POSTGRES_CONNECTION_URI=postgres://postgres:postgres@postgres:5432/postgres
ADBC_OUTBOX_TABLE_AUTO_CREATE=true
ADBC_NODES_TABLE_AUTO_CREATE=true
Demo OnlyCopyCopied!
The docker-compose.yml
will load this configuration into the adbc
container as environment variables.
Run docker compose up
to build and run the Docker containers:
docker compose up --build
CopyCopied!
Ping the database connector health endpoint to verify everything works correctly.
curl localhost:2259/health
CopyCopied!
The response should look similar to the following:
{"ably":{"status":"up"},"nodes_table":{"status":"up"},"outbox_table":{"status":"up"},"postgres":{"status":"up"}}
CopyCopied!
Verify connection using Docker
Verify your connection to the Ably network now you have a PostgreSQL database and Database Connector instance running. The following examples show how to write a record to the outbox table and confirm its publication.
Subscribe to a channel named foo
using Server-Sent Events:
curl -s -u "<loading API key, please wait>" "https://realtime.ably.io/sse?channel=foo&v=1.1"
Demo OnlyCopyCopied!
Add a record to the outbox table in your PostgreSQL database. Use the psql tool in the Postgres container to execute an SQL statement against the database:
docker exec -it -u postgres adbc-postgres \
psql postgres://postgres:postgres@postgres:5432/postgres \
-c "INSERT INTO outbox (mutation_id, name, channel, data, headers) \
VALUES ('1', 'test', 'foo', '{}', '{}');"
CopyCopied!
The Database Connector detects and publishes newly inserted records as messages to the specified channel in the record.
You will receive a response similar to the following, indicating the successful receipt of the event over your SSE connection to Ably:
id: 108GsR8ewBVHhJ@1700069266489-0
event: message
data: {"id":"1","connectionId":"CaqkrZ2N_0","timestamp":1700069266050,"encoding":"json","extras":{"headers":{"x-ably-models-event-uuid":"1"}},"channel":"foo","data":"{}","name":"test"}
CopyCopied!
Run a new database
Use Railway with a PostgreSQL plugin instead of using your existing database.
Setup in Railway
Create a free Railway account, install the Railway CLI and log in:
npm i -g @railway/cli
railway login
CopyCopied!
Create a new Railway project and link it to your project directory:
railway init
# Enter a project name, e.g., "adbc-test"
railway link
CopyCopied!
Add the PostgreSQL plugin to your project:
railway add --plugin postgresql
CopyCopied!
Create a new Dockerfile using the Database Connector as the base image. Railway requires a Dockerfile to define the application for deployment:
echo "FROM ghcr.io/ably-labs/adbc:latest" > Dockerfile
CopyCopied!
Use railway up
to deploy your application. This command will build the Database Connector container image from the Dockerfile in your project root and deploy it to your Railway project:
railway up --detach
CopyCopied!
Configure in Railway
Open your project in the Railway console:
railway open
CopyCopied!
Select the adbc-test
service, then navigate to Variables → RAW Editor. Paste the following variables into this section:
ADBC_ABLY_API_KEY=<loading API key, please wait>
ADBC_POSTGRES_CONNECTION_URI=${{Postgres.DATABASE_URL}}
ADBC_OUTBOX_TABLE_AUTO_CREATE=true
ADBC_NODES_TABLE_AUTO_CREATE=true
Demo OnlyCopyCopied!
Railway will restart the adbc-test
service with the newly applied configuration.
Verify connection using Railway
Verify your connection to the Ably network now you have a PostgreSQL database and Database Connector instance running. The following examples show how to write a record to the outbox table and confirm its publication.
Subscribe to a channel named foo
using Server-Sent Events:
curl -s -u "<loading API key, please wait>" "https://realtime.ably.io/sse?channel=foo&v=1.1"
Demo OnlyCopyCopied!
Add a record to the outbox table in your PostgreSQL database. Use the Railway CLI to execute an SQL statement against the database:
railway connect postgres
railway=# INSERT INTO outbox (mutation_id, name, channel, data, headers)
VALUES ('1', 'test', 'foo', '{}', '{}');
CopyCopied!
The Database Connector detects and publishes newly inserted records as messages to the specified channel in the record.
You will receive a response similar to the following, indicating the successful receipt of the event over your SSE connection to Ably:
id: 108GsR8ewBVHhJ@1700069266489-0
event: message
data: {"id":"1","connectionId":"CaqkrZ2N_0","timestamp":1700069266050,"encoding":"json","extras":{"headers":{"x-ably-models-event-uuid":"1"}},"channel":"foo","data":"{}","name":"test"}
CopyCopied!
Configuration options
You can use one of the following methods to override the default values of configuration options:
Environment variables
Specify configuration options as environment variables by using the following method:
- Capitalize each option.
- Separate each word with underscores.
- Prefix each option with
ADBC_
to denote a namespace and avoid conflicts with other variables.
Set your Ably API key using an environment variable:
docker run -it -e ADBC_ABLY_API_KEY=<loading API key, please wait> ghcr.io/ably-labs/adbc:latest
Demo OnlyCopyCopied!
The following is an example of reading the API key from a .env
file:
echo “ADBC_ABLY_API_KEY=<loading API key, please wait>” >> adbc.env
docker run -it --env-file=adbc.env ghcr.io/ably-labs/adbc:latest
Demo OnlyCopyCopied!
YAML
Specify configuration options as YAML by using the following method:
- Use camel case for each option.
- Name the file
adbc.yaml
or.adbc.yaml
. - Host it in the working directory, the
$HOME
directory, or specify its path using--config
.
Set your Ably API key using a YAML file:
ably:
apiKey: "<loading API key, please wait>"
Demo OnlyCopyCopied!
Ensure the YAML file is accessible to the application by mounting it inside the container:
docker run -it --volume "$(pwd)/adbc.yaml:/adbc.yaml:ro" ghcr.io/ably-labs/adbc:latest
CopyCopied!
Verify the YAML file is mounted inside the container by using the following method:
version: '3'
services:
adbc:
image: ghcr.io/ably-labs/adbc:latest
volumes:
- ./adbc.yaml:/adbc.yaml:ro # mount yaml config file
CopyCopied!
Command line flags
Specify configuration options using CLI:
- Use snake case for each option.
- Prefix each with
--
.
Set your Ably API key using CLI:
docker run -it ghcr.io/ably-labs/adbc:latest --ably-api-key=<loading API key, please wait>
Demo OnlyCopyCopied!
Commonly used configuration options
Use --help
to view the complete set of configuration options available on the connector:
docker run -it --entrypoint="/adbc" ghcr.io/ably-labs/adbc:latest --help
CopyCopied!
The following table provides descriptions for the most commonly used configuration options:
Option | Description |
---|---|
ADBC_ABLY_API_KEY | Your Ably API key. |
ADBC_POSTGRES_CONNECTION_URI | The full connection URI of your Postgres database. |
ADBC_POSTGRES_HOST | Your Postgres database host name as an alternative to providing the CONNECTION_URI . |
ADBC_POSTGRES_PORT | Your Postgres database port number as an alternative to providing the CONNECTION_URI . |
ADBC_POSTGRES_DATABASE | Your Postgres database name as an alternative to providing the CONNECTION_URI . |
ADBC_POSTGRES_USER | Your Postgres database user as an alternative to providing the CONNECTION_URI . |
ADBC_POSTGRES_PASSWORD | Your Postgres database user password as an alternative to providing the ADBC_POSTGRES_CONNECTION_URI . |
--config | Can only be specified as a CLI flag and enables you to override the path to a YAML configuration file. |
ADBC_ENV | An environment descriptor (either development or production). development pretty-prints log output with logging enabled at debug level and above. production logs output in JSON format with logging enabled at info level and above. |
ADBC_LOG_LEVEL | Specifies the log level to use (one of: debug, info, warn, error, fatal, panic) and overrides any presets from ADBC_ENV . |
ADBC_OUTBOX_TABLE_TABLE_SCHEMA | Configures the database schema of the outbox table. |
ADBC_OUTBOX_TABLE_TABLE_NAME | Configures the name of the outbox table. |
ADBC_OUTBOX_TABLE_AUTO_CREATE | Configures the application to create the outbox table if it doesn’t already exist on startup. |
ADBC_NODES_TABLE_TABLE_SCHEMA | Configures the database schema of the nodes table. |
ADBC_NODES_TABLE_TABLE_NAME | Configures the name of the nodes table. |
ADBC_NODES_TABLE_AUTO_CREATE | configures the application to create the nodes table if it doesn’t already exist on startup. |
ADBC_HEALTH_ADDRESS | Configures the TCP address for the server to listen on in the form host:port. |
ADBC_POLL_FIXED_RATE | If true, the application polls the outbox table at a fixed rate given by ADBC_POLL_INTERVAL (default 1 second). If false, the application uses a trigger with LISTEN/NOTIFY to poll for records only when the data in the outbox changes. |