Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

September 1, 2025

AWS DynamoDB Shell - ddbsh

DynamoDB Shell (ddbsh) commands in AWS


An open-source command line interface (CLI) and interactive shell to run SQL-like DDL and DML commands on Amazon DynamoDB (DDB). And these SQL commands will be automatically translated to DynamoDB queries. DynamoDB Shell is written in C++ and uses the DynamoDB API through the AWS DynamoDB SDK.

For installation and building the DynamoDB Shell tool, please refer to the project’s GitHub page.

$ ddbsh
help update;
help;
HELP - provide help in ddbsh

   HELP <keyword> [keyword [keyword ...]]

      Provides help about the specified keyword, or statement.

      HELP ALTER TABLE
      HELP BACKUP
      HELP BEGIN
      HELP COMMIT
      HELP CONNECT
      HELP CREATE
      HELP DELETE
      HELP DESCRIBE BACKUP
      HELP DESCRIBE
      HELP DROP BACKUP
      HELP DROP TABLE
      HELP DROP
      HELP EXPLAIN
      HELP INSERT
      HELP REPLACE
      HELP RESTORE
      HELP ROLLBACK
      HELP SELECT
      HELP SHOW BACKUPS
      HELP SHOW CREATE TABLE
      HELP SHOW LIMITS
      HELP SHOW VERSION
      HELP SHOW
      HELP UPDATE
      HELP UPSERT

connect us-east-2;
show version;
show limits;
show tables;
begin;
commit;
rollback;
describe dynamodb_tab;
show create table ddbsh_demo;

CREATE TABLE commands in DynamoDB shell

CREATE TABLE [IF NOT EXISTS][NOWAIT] <name>  ( attribute_name, attribute_type [,...] )
   primary_key billing_mode_and_throughput
   [gsi_list] [lsi_list] [streams] [table_class] [tags] ;
create table ddbsh_demo (id number) primary key (id hash);
create table ddbsh_demo2 (pk number, rk number) primary key (pk hash, rk range);
create table ddb_tab ( a number, b number ) primary key ( a hash ) gsi ( gsione on (b hash) projecting all);
create table ddb_table ( id string, name string ) primary key ( id hash ) billing mode provisioned ( 5 rcu, 5 wcu ) gsi ( namegsi on (name hash) projecting all billing mode provisioned ( 5 rcu, 5 wcu ));
create table if not exists nowait ddbsh_demo3 (id string, accttype string, balance number) primary key (id hash, accttype range) billing mode provisioned (20 rcu, 20 wcu) gsi (balancegsi on (accttype hash, balance range) projecting all billing mode provisioned (20 rcu, 20 wcu)) stream (both images);

ALTER TABLE commands on DynamoDB tables using DDB shell

alter table dynamodb_tab set billing mode on demand;
alter table dynamodb_tab set billing mode provisioned (200 RCU, 300 WCU);
alter table dynamodb_tab set table class standard infrequent access;
alter table dynamodb_tab set table class standard;
alter table dynamodb_tab set stream (new image);
alter table dynamodb_tab set stream disabled;
alter table dynamodb_tab set pitr enabled;
alter table dynamodb_tab set pitr disabled;
alter table dynamodb_tab set deletion protection disabled;
alter table dynamodb_tab set deletion protection enabled;
alter table ddb_tab (x number, y number) create gsi xygsi on (x hash, y range) projecting all billing mode provisioned (20 RCU, 40 WCU);
alter table ddb_tab (x number, y number) create gsi xykeysgsi on (x hash, y range) projecting keys only billing mode provisioned (22 RCU, 41 WCU);
alter table ddb_tab (x number, y number, a number) create gsi xyagsi on (x hash) projecting include (y, a) billing mode provisioned (25 RCU, 50 WCU);
alter table ddb_tab set billing mode provisioned (20 rcu, 30 wcu) update gsi (pqgsi set billing mode provisioned (20 rcu, 30 wcu), xykeysgsi set billing mode provisioned (3 rcu, 5 wcu));
alter table ddb_tab drop gsi xygsi;
alter table ddbsh_demo add replica us-east-2;
alter table ddbsh_demo add replica us-west-2 table class standard infrequent access;
alter table ddbsh_demo drop replica us-east-2;
alter table ddbsh_demo set ttl (created_epoch_time);
alter table ddbsh_demo set ttl disabled;

If you want to add new column(s), there is NO command like alter table table-name add column column-name, you have to use INSERT or UPDATE or UPSERT commands with values for new columns (along with existing columns).

DROP TABLE commands on Amazon DynamoDB tables

drop table ddbsh_demo;
drop table if exists ddb_tab;

SELECT queries/commands on DynamoDB tables using DynamoDB shell

SELECT [CONSISTENT] attribute_list | * FROM <table>[.<index>] [WHERE where_clause] [return_clause | ratelimit] ]
select * from ddbsh_demo;
select * from ddbsh_demo where string_col = "available";
select * from ddbsh_demo where a = 5;
select id, a from ddbsh_demo where a > 3 and a < 5;
select * from ddbsh_demo where a between 2 and 5;
select * from ddbsh_demo where rangekey in (3, 9, 4, 7);
select consistent * from ddbsh_demo where attribute_exists(c) and a != 8;
select * from ddbsh_demo where attribute_exists(c) or a = 33;
select * from ddbsh_demo where attribute_exists(c);
select * from ddbsh_demo where not attribute_exists(amount);
select * from ddbsh_demo where attribute_type(b, string);
select * from ddbsh_demo where attribute_type(id, number) return total;
select * from ddbsh_demo where begins_with(b, "co");
select * from ddbsh_demo where contains(pk, "at");
select a, b, c from ddbsh_demo where attribute_type(b, string) or (a = 5 and c = "coffee");
select * from ddbsh_demo where size(id) < 5;
select * from ddbsh_demo where v.c = true;
select * from ddbsh_demo where v.b[1] = 11;
select * from ddbsh_demo.gsi1;
select * from ddb_tab.gsi2 where name = "Brutus";
select * from ddbsh_demo where a between 2 and 5 WITH RATELIMIT (5 RCU);
select * from ddbsh_demo where a between 2 and 5 WITH RATELIMIT (2 RCU, 2 WCU);

INSERTing records INTO DDB tables with DynamoDB shell

insert into ddbshell (custid, name) values (103, "Charlie");
insert into ddbshell (custid, name) values (101, "Alice"), (102, "Bob");
insert into ddbshell (pk, rk, x, y, z) values (4, "three", 11, 12, 13);
insert into ddbshell (id, v) values (3, 4), (4, "a string value"), (5, {a: 4, b: [10, 11, 12], c: true, d: {x: 10, y: 10}});
insert into ddb_tab (a, b) values ( 1, 2 ), (2, 3), (3, 4) with ratelimit ( 2 wcu );

REPLACE INTO <table> ( column [, column ...] ) VALUES ( values ) [ratelimit]
replace into ddbshell (pk, rk, ins) values (12, "nonexistant", "inserted");

UPDATE commands on Amazon DynamoDB tables using DDBShell

update ddbsh_demo set b = 15 where a = 5;
update ddb_tab age= 14 where name = "Brutus";
update ddbsh_demo set active = True where custid = 111;
update ddbsh_demo set newattr = 14 where pk = 1 and rk = "one";
update ddbsh_demo set newattr = if_not_exists(y, 3) + 6 where pk = 1;
update ddb_tab set balance = balance + 100 where id = "Satya" and accttype = "Savings";
update ddbsh_demo set z = 14, v.b[1] = 13 where id = 5;
update ddb_tab set updated = true with ratelimit ( 10 rcu, 5 wcu );
update ddbsh_demo remove c where a = 5; (works like alter table table-name drop column column-name)

UPSERT <name> SET <upsert_set> [where clause] [ratelimit]
upsert ddbsh_demo set op = "upsert" where pk = 2;

Deleting records from DynamoDB tables using DDB shell

delete from ddb_tabble where name = "Brutus";
delete from ddbsh_demo where pk = 1 and rk = "one";

Backup and Restore of AWS DynamoDB tables

backup table dynamodb_tab called dynamodb-table-backup;
show backups;
describe backup "arn:aws:dynamodb:us-east-2:1234567890:table/dynamodb_tab/backup/1681480720-2a11c134";
drop backup "arn:aws:dynamodb:us-west-2:1234567890:table/backup_test/ddbsh_demo/1681480720-2a11c134";

restore table "dynamodb_tab" from backup "arn:aws:dynamodb:us-east-2:1234567890:table/dynamodb_tab/backup/1681480720-2a11c134";
RESTORE TABLE new table name FROM old table name TO PITR "YYYY-MM-DDTHH:MM:SSZ";

Getting EXPLAIN plan for DynamoDB (shell) queries

explain select * from ddbsh_demo2;
explain select * from dynamodb_tab where attribute_type(b, string) or (a = 5 and c = "coffee");
explain create table ddbsh_demo (id string, name string) primary key (id hash) billing mode provisioned (5 rcu, 5 wcu) gsi (namegsi on (name hash) projecting all billing mode provisioned (5 rcu, 5 wcu));
explain alter table ddb_tab add replica us-west-2;
explain alter table ddbsh_demo (v number) create gsi gsi_v on (v hash) projecting all billing mode provisioned (10 rcu, 20 wcu);
explain update ddbsh_demo set z = 14, v.b[6] = 13 where id = 5;
explain update ddb_table.zipgsi set state = "CA" where zip = "90210" with ratelimit (10 rcu, 20 wcu);
explain delete from ddb_tab where rk = "two";


Related AWS Cloud Articles:  AWS Cloud Database/DBA Interview Questions

May 14, 2023

30 Top Cloud DBA Database Interview Questions

AWS Cloud DBA/Database Interview Questions - Part 1


Q1. What are the advantages of cloud computing?
answer:

Q2. What is the database storage in AWS cloud? where are the RDS instance datafiles stored?
answer:

Q3. What database metrics will you be monitoring?
answer:

Q4. How will you check database performance issues/bottlenecks in AWS console?
answer:

Q5. Which EC2 instance type/class is the best for databases?
answer:

Q6. How to enforce the security of RDS instances? 
answer:

Q7. How to encrypt data-at-rest in the cloud?
answer:

Q8. How to encrypt data-in-transit / data-in-motion in the cloud?
answer:

Q9. What is the difference/advantages between RDS instance and database running on EC2?
answer:

Q10. What is the difference between "General purpose" and "Provisioned IOPS" storage?
answer:

Q11. What is the advantage of Multi-AZ instance in AWS?
answer:

Q12. How do we export and import data/snapshots from production to non-production environments in different AWS accounts?
answer:

Q13. Please explain the main differences between RDS MySQL and AWS Aurora?
answer:

Q14. What are the main differences between RDS PostgreSQL and Amazon Aurora?
answer:

Q15. How to setup MySQL replication between two Amazon Aurora MySQL clusters?
answer:

Q16. How to setup logical replication between two AWS Aurora Postgres clusters?
answer:

Q17. How can you migrate data from 
MySQL RDS and Aurora MySQL?
answer:

Q18. How will you update the database parameter value of (Oracle) RDS instance?
answer:

Q19. 
answer:

Q20. 
answer:

Related DBA Articles: PostgreSQL/Aurora Postgres DBA Interview Questions   AWS DBA Database Interview Questions - Part 2

April 24, 2023

Best 30 AWS Cloud Database Interview Questions

AWS Cloud DBA/Database Interview Questions - part 2


Q21. How to perform point-in-time recovery (PITR) of the RDS database?
answer:

Q22. What is the advantage and dis
advantage of the Aurora Global cluster?
answer:

Q23. How to create and maintain AWS resources?
answer:

Q24. What is Terraform/IaC (Infrastructure as Code)? How do you manage Cloud resources using Terraform?
answer:

Q25. What are the providers/modules in Terraform?
answer:

Q26. What are the different storage classes in S3 storage? 
answer:

Q27. What are latency and throughput in Cloud computing?
answer:

Q28. What are the Cloudwatch Metrics you are monitoring for your databases?
answer:

Q29. What is Aurora serverless option? what are the advantages of Aurora serverless?
answer:

Q30. How do you encrypt an unencrypted RDS database instance?
answer:

Q31. What is Host Based Access (HBA) control in Postgres databases? Do we need to enable/use HBA in AWS PostgreSQL RDS instances?
answer:

Q32. 
answer:

Q33. 
answer:

Related Database Articles:  Cloud DBA - AWS Database Interview Questions - Part 1

March 5, 2023

pglogical replication between PostgreSQL RDS/Aurora Postgres databases

Replication setup between Amazon Aurora Postgres/PostgreSQL RDS instances using pglogical


pglogical extension provides additional capabilities that aren’t available in native PostgreSQL.
Publish/subscribe model to migrate critical workloads.
Used to migrate data between PostgreSQL versions and replicate data between two or more PostgreSQL instances regardless of where those instances are located.
Used to migrate whole clusters, specific tables, and specific rows/columns.

Limitations of pglogical:
  • Needs superuser privileges
  • UNLOGGED/TEMPORARY tables/views can't be replicated
  • Primary key or unique key is mandatory
  • Sequence data is not replicated
  • Automatic DDL replication is not supported (other than TRUNCATE)
  • It is not recommended to add additional UNIQUE constraints other than the PRIMARY KEY
  • Unique constraints must not be deferrable
  • Large objects (LOBs) are not replicated
  • Foreign key constraints aren’t enforced during the replication process

Nodes - PostgreSQL instances
Providers and Subscribers - roles taken by Nodes
Replication Set - a collection of tables; used to control which tables in the database are replicated and which actions on those tables are replicated.

Step #1: Parameter changes, on both source/target databases

Cluster parameter group   -> some parameters are static, needs reboot of cluster
rds.logical_replication = 1
shared_preload_libraries = ***, pglogical
wal_level = 'logical'
max_worker_processes = 10    # one per database needed on provider node and one per node needed on subscriber node
max_replication_slots = 10  # one per node needed on provider node
max_wal_senders = 10        # one per node needed on provider node
track_commit_timestamp = on # needed for last/first update wins conflict resolution

Instance parameter group  -> parameter is static, needs reboot of instance
shared_preload_libraries = ***, pglogical

If debugging/logging more information required, please set below parameters.
log_min_messages=debug1
log_min_error_statement=debug1
log_error_verbosity=verbose
log_statement = 'all'

Step #2: Create user & extension, on both source/target databases

create user pguser with password 'pguser123';
grant rds_superuser to pguser;
 
create extension pglogical;

Step #3: Create Provider node (source database)

select pglogical.create_node(node_name := 'provider1', dsn := 'host=providerhost port=5432 dbname=db');
select pglogical.create_node(node_name :='node1', dsn := 'host=<db-endpoint> port=5432 dbname=<dbname> user=pguser password=pguser123);
select pglogical.create_node(node_name := 'node1', dsn := 'host= lab-pg14.xxxxxx.us-east-2.rds.amazonaws.com port=5432 sslmode=require dbname=cluster1 user=pguser password=xxxx);

select pglogical.drop_node(node_name := 'node1234');         # if needed to recreate node

Step #4: Create replication set and add tables to replication set, on source database

There are 3 preexisting replication sets named "default", "default_insert_only" and "ddl_sql".

select pglogical.create_replication_set(set_name name, replicate_insert bool, replicate_update bool, replicate_delete bool, replicate_truncate bool)
select pglogical.create_replication_set('shared_repl', true, true, true, true );
select pglogical.create_replication_set('test_repl_set');
select pglogical.create_replication_set(set_name := 'dml-replication-set', replicate_insert := TRUE, replicate_update := TRUE, replicate_delete := TRUE, replicate_truncate := FALSE);
select pglogical.alter_replication_set(set_name name, replicate_inserts bool, replicate_updates bool, replicate_deletes bool, replicate_truncate bool)

select pglogical.replication_set_add_all_tables('default', ARRAY['public']);      # adds all tables from schema 'public' to 'default' replication set
select pglogical.replication_set_add_all_tables('replication_set', ARRAY['catalog2'], true);
select pglogical.replication_set_add_all_tables('test_repl_set', ARRAY['schematest']);

select pglogical.replication_set_add_table(set_name name, relation regclass, synchronize_data boolean, columns text[], row_filter text)
select pglogical.replication_set_add_table('default_insert_only', 'public.pgbench_history');
select pglogical.replication_set_add_table(set_name := 'replication_set', relation := 'test', synchronize_data := true );
select pglogical.replication_set_add_table(set_name := 'test_repl_set', relation := 'schematest.employees');

select pglogical.replication_set_add_all_sequences(set_name name, schema_names text[], synchronize_data boolean)
select pglogical.replication_set_add_all_sequences(set_name := 'default', schema_names := '{public}'::text[], synchronize_data := true)
select pglogical.replication_set_add_sequence(set_name name, relation regclass, synchronize_data boolean)
select pglogical.synchronize_sequence( seqoid ) from pglogical.sequence_state;
select seqoid::regclass from pglogical.sequence_state"

select pglogical.drop_replication_set('default');     # if needed to recreate replication set
select pglogical.replication_set_remove_table(set_name name, relation regclass)
select pglogical.replication_set_remove_table('default', 'listing');

Step #5: Create subscription, on target database

select pglogical.create_subscription(subscription_name := 'subscription1', provider_dsn := 'host=providerhost port=5432 dbname=db');
select pglogical.create_subscription(
    subscription_name := 'west_sub',
    provider_dsn := 'host=testdb.cluster-****.us-west-2.rds.amazonaws.com port=5433 dbname=postgres user=pguser password=pguser123',
    replication_sets := ARRAY['default'],
    synchronize_data := false,
    forward_origins := '{}' );
select pglogical.create_subscription(subscription_name := 'cluster2_sub', provider_dsn := 'host=lab-pg14.xxxxxx.us-east-1.rds.amazonaws.com port=5434 sslmode=require dbname=cluster1 user=pguser password=xxxx', replication_sets := ARRAY['default'], synchronize_data := true, forward_origins := '{}' );

select pglogical.alter_subscription_disable('subscription_name');
select pglogical.alter_subscription_disable(subscription_name) from pglogical.subscriptions where writer = 'name_of_writer';
select pglogical.alter_subscription_enable('west_to_east2_sub');
select pglogical.alter_subscription_enable('west_to_east2_sub', true);
select pglogical.drop_subscription('subscription1');

select pglogical.alter_subscription_skip_changes_upto('subscription_name','the_target_lsn');
select pglogical.wait_for_subscription_sync_complete('subscription1');

Step #6: Commands to check pglogical replication status

select pglogical.show_subscription_status('west_sub');
select pglogical.show_subscription_table(subscription_name name, relation regclass)
select pglogical.show_subscription_table('west_sub','listing');

select * from pglogical.subscription;
select subscription_name, status from pglogical.show_subscription_status() order by subscription_name;
select * from pglogical.show_subscription_status();

select * from pg_replication_slots;
select slot_name, slot_type, active from pg_replication_slots;
select slot_name from pg_replication_slots where active='f'; # inactive replications slots
select redo_lsn, slot_name, restart_lsn, round((redo_lsn-restart_lsn)/1024/1024/1024, 2) AS GB_behind from pg_control_checkpoint(), pg_replication_slots;

To remove replication slot, if required:
select pg_terminate_backend(23593);
select pg_drop_replication_slot('pgl_postgres_node_east_east_to_west_sub');

select * from pg_stat_replication_slots;
select slot_name, spill_txns, spill_count, spill_bytes from pg_stat_replication_slots;
select stream_txns, stream_count, stream_bytes from pg_stat_replication_slots;

select * from pg_stat_replication;
select pid, usename, application_name, client_addr, state, sync_state,
       pg_wal_lsn_diff(sent_lsn, write_lsn) as write_lag, pg_wal_lsn_diff(sent_lsn, flush_lsn) as flush_lag, 
       pg_wal_lsn_diff(sent_lsn, replay_lsn) as replay_lag, pg_wal_lsn_diff(sent_lsn, replay_lsn) as total_lag 
from pg_stat_replication;

select * from pglogical.depend;
select * from pglogical.node order by node_name;
select * from pglogical.local_node;
select * from pglogical.pglogical_node_info();

select * from pglogical.local_sync_status;
SELECT * FROM pglogical.local_sync_status WHERE NOT sync_status = 'r';
SELECT sync_status FROM pglogical.local_sync_status WHERE sync_nspname = 'public' AND sync_relname = 'example';
sync_kind
i initial sync
d data sync
f full sync
s structure sync

sync_status
\0: No sync
i: initial - Ask for sync
r: sync done
d: data sync
c: constraints sync
s: structure sync
u: Catching up
w: Table sync is waiting (to get OK from main thread)
y: sync finished at LSN

select * from pglogical.replication_set order by set_name;
select * from pglogical.replication_set_table;
select * from pglogical.replication_set_table where set_reloid::text like '%listing%';
select * from pglogical.queue;

select * from pg_publication;
select * from pg_publication_tables;
select * from pg_stat_subscription;

select pglogical.alter_subscription_resynchronize_table(subscription_name name, relation regclass)
select pglogical.alter_subscription_resynchronize_table('east2_to_west2_sub', 'venue');


Related PostgreSQL Articles: 30 Best PostgreSQL DBA Interview Questions  pg_restore - PostgreSQL database/backup restore tool


November 25, 2022

40 Best (Aurora) PostgreSQL Interview Questions - part 2

40 Top PostgreSQL/Aurora Postgres DBA Interview Questions - Part2


Q21. Which are the methods PostgreSQL provides to create a new database?
AnswerPostgreSQL provides the following methods to create a new database:
  • Using CREATE DATABASE, an SQL command
  • Using created a command-line executable

Q22. How do you delete the database in PostgreSQL?
Answer: We can delete the database by using any one of the below options:
  • Using DROP DATABASE, an SQL command
  • Using dropdb a command-line executable

Q23. What is table partitioning in Postgres?
Answer:

Q24. Explain PostgreSQL Inheritance. How does PostgreSQL implement table inheritance? What is the use of it?
Answer:

Q25. What types of indexes are supported in PostgreSQL?
Answer:

Q26. How to take the backup of PostgreSQL?
Answer:

Q27. What is a CTID of PostgreSQL?
Answer:

Q28. What are the different datatypes in PostgreSQL?
Answer:

Q29. How are stats/statistics updated in PostgreSQL?
Answer:

Q30. How to start a PostgreSQL database server?
Answer:

Q31. How To stop the PostgreSQL database server?
Answer:

Q32. How to check the status of the PostgreSQL server running or not?
Answer:

Q33. How to get a list of databases in a PostgreSQL cluster?
Answer:

Q34. How to take differential or incremental backup of PostgreSQL database?
Answer: There is NO utility for differential or incremental backups.

Q35. How to upgrade the Postgres database server?
Answer:

Q36. How crash recovery works in Postgres?
Answer:

Q37. What are the default tablespaces created in Postgres?
Answer:

Q38. What are the levels of isolation?
Answer:

Q39. How to setup logical replication between two AWS Aurora Postgres clusters?
Answer:

Q40. 
Answer:


Related Postgres Articles: PostgreSQL/Aurora Postgres Database Interview Questions Part1