v2.3 series release notes
Included here are the release notes for all releases in the v2.2 release series.
v2.3.0.0
Released: September 8, 2020 (2.3.0.0-b176).
Downloads
Binaries
Docker
docker pull yugabytedb/yugabyte:2.3.0.0-b176
Features and changes
YSQL
- Fix OOM when running large
COPY TO
statements by creating new memory context for the loop over retrieved rows and resetting it after processing each row. #5205 - Support transactional batch size for
COPY FROM
command with OOM fix. Batch sizes can be passed in withROWS_PER_TRANSACTION
in theCOPY OPTION
syntax. For an example, see Import a large table using smaller transactions #2855 #5453 - For index backfill flags, use better default values. Set
index_backfill_rpc_timeout_ms
default from60000
to30000
and changebackfill_index_timeout_grace_margin_ms
default from50
to500
. #5494 - Remove spurious error message "0A000: Alter table is not yet supported" from
CREATE OR REPLACE VIEW
. #5071 - Prevent consistency violations when a partitioned table has foreign key constraints due to erroneous classification as a single-row transaction. #5387
- Fix restore from a distributed backup fails for tables using
SPLIT INTO
without a primary key. #4993 - Fix wrong result by avoiding pushdown of
UPDATE
statement with theRETURNING
clause. #5366 - Improve error message when
UPDATE
changes partition so its clear what went wrong, fixedyb_pg_foreign_key
pg_regress
test for semantic merge conflict when updating primary keys and row-level partitioning. #659 #5179 #5310 - Allow
UPDATE
statement to change primary key columns. #659 - Support row-level partitioning. #5179
- Block usage of
TABLEGROUP
with theSPLIT
clause. ForCREATE TABLE
, usage is blocked in the grammar. ForCREATE INDEX
, ifNO TABLEGROUP
was provided, then presplitting for the index is allowed; otherwise, an error is issued. #5352 - Buffered operations may share single RPC with read operation. Reducing the number of RPC calls speeds up
CREATE TABLE
statement. For example, the total number of RPC calls is dropped from 66 to 58 for a simple table likeCREATE TABLE t(k INT PRIMARY KEY)
. #5177 - Add new
ysqlsh
describe metacommands for tablegroups:\dgr[+] [grpname]
to describe tablegroups,\dgrt[+] [grpname]
lists all tables/indexes within the specified tablegroup (or within all tablegroups ifgrpname
is not specified), and\d <table_name>
is modified to include tablegroup information in the footer, if any. #5088 - Add
ALTER TABLEGROUP
statements to supportALTER TABLEGROUP tablegroup_name RENAME TO ...
andALTER TABLEGROUP tablegroup_name OWNER TO ...
. Also changespg_tablegroup
entry corresponding totablegroup_name
to properly reflect newgrpname
andgrpowner
(if the user has proper permissions or ownership to issue theALTER TABLEGROUP
statement. #5249 - Add support for indexes to opt out of tablegroups (
NO TABLEGROUPS
) or select their own tablegroup (TABLEGROUP group_name
). #5293 - For
ysql_dump
, enableserializable-deferrable
mode by default. Add new--no-serializable-deferrable
flag to disable the default mode. #5128 - Support
ALTER COLUMN type
for variants that don't require on-disk changes. Specifically only allowingALTER COLUMN type
forvarch(n)
andvarbit(n)
. For example, changing column type fromvarchar(50)
tovarchar(255)
. #4424 - Enable
USING
clause inDELETE
statement andFROM
clause inUPDATE
statement. #738 #5262 - Support GRANT/REVOKE/ALTER DEFAULT PRIVILEGES for tablegroups. #5087 #5160
- Implement tablegroup query layer changes. #4525
- Replace "Foreign Scan" with "Seq Scan" for
EXPLAIN
statement. #2076 - Fix
initdb
when index backfill is enabled. #5027 - Properly handle empty delete with backfill by sending appropriate messages to the client, including setting it
skipped
androws_affected_count
to0
. #5015 DROP INDEX
statement invalidates table cache entry for the index table, but should invalidate the table cache entry for the indexed table. #4974yb-admin create_database_snapshot
command should not requireysql.
prefix for database name. #4991- For non-prepared statements, optimize
pg_statistic
system table lookups and update debugging utilities. #5051 - Correctly show beta feature warnings by default. #5322
YCQL
- For
WHERE
clause in theCREATE INDEX
statement, return aNot supported
error. #5363 - Fix TSAN issue in partition-aware policy for C++ driver 2.9.0-yb-8 (yugabyte/cassandra-cpp-driver). #1837
- Support YCQL backup for indexes based on JSON-attribute. #5198
- Correctly set
release_version
forsystem.peers
queries. #5407 - Reject
TRUNCATE
operations whenycql_require_drop_privs_for_truncate
flag is enabled. When enabled,DROP TABLE
permission is required to truncate a table. Default isfalse
. #5443 - Fix missing return statement in error case of the
SetPagingState
method instatement_params.cc
. #5441 - Enable backfilling of transactional tables by default. #4708
- Fix
ycqlsh
should return a failure when known that the create (unique) index has failed. #5161
Core database
- Fix core dump related to DNS resolution from cache for Kubernetes universes. #5561
- Fix yb-master fails to restart after errors on first run. #5276
- Show better error message when using
yugabyted
and yb-master fails to start. #5304 - Disable ignoring deleted tablets on load by default. #5122
- [CDC] Improve CDC idle throttling logic to reduce high CPU utilization in clusters without workloads running. #5472
- For the
server_broadcast_addresses
flag, provide default port if not specified. #2540 - [CDC] Fix CDC TSAN destructor warning. #4258
- Add an API endpoint to download root certificate file. #4957
- Do not load deleted tables and tablets into memory on startup. #5122
- Set the follower lag for leaders to
0
by resetting timestamp to the maximum value when a peer becomes a leader and when the peer loses leadership. #5502 - Flow keyspace information from yb-master to yb-tserver. #3020
- Implement meta cache lookups throttling to reduce unnecessary thrashing. #5434
- Fix
rpcz/statements
links inyb-tserver
Web UI whenpgsql_proxy_bind_address
andcql_proxy_bind_address
are0.0.0.0
. #4963 DumpReplayStateToStrings
should handle too many WAL entries scenario. Also, log lines only display fields critical for debugging and do not show customer-sensitive information. #5345- Find the difference between the replica map and consensus state more quickly using a map lookup. #5435
- Improve failover to a new master leader in the case of a network partition or a dead tserver by registering tablet servers present in Raft quorums if they don't register themselves. Also, mark replicas that are in
BOOTSTRAPPING
/NOT_STARTED
state with its true consensus information instead of marking it as aNON_PARTICIPANT
. #4691 - Add YB-Master UI changes to support tablegroups and colocation. #5086
- Adds a column to the user and index tables to display the YSQL OID information about the colocation parent table (if any).
- For tablegroups, it displays the OID of the table's tablegroup (same as the OID present in
pg_tablegroup
as well as thereltablegroup
column ofpg_class
for that relation). - For colocated databases, it displays the OID of the database.
- Also, creates a table to display information about the parent tables. Wrap the names and uuid of the parent tables. Display YSQL OIDs as explained above.
- Fix
MetaCache::TAbleData::stale
is not getting reset back tofalse
. #5245 - Do not crash yb-tserver when the
op_id
of the last WAL file is less than the committedop_id
. #1560 - Add
CREATE TABLEGROUP
andDROP TABLEGROUP
flow. #4525 - Add
yb-admin list_snapshots
SHOW_DELETED
flag to show deleted snapshots that are still retained in memory. #5332 - Move read path functions to a separate file. #4944
- Wait for
yb_client_admin_operation_timeout_sec
for table creation inyb-admin import_snapshot
task. #5295 - Create a cache for system-wide queries. #5043
- Annotate access to
enable_collect_cdc_metrics
flag and prevent TSAN warnings. #5303 - Raise yb-master and yb-tserver process ulimits to hard limits when appropriate. #4943
- Add a pre-flush step, triggered by yb-master, to snapshot operations to minimize the amount of data being flushed in sync. #4718
- Use Strand to remove intents and avoid wasting threads. #5265
- Add metric severity level [PORT] and sever-side level-based filtering for metrics and prometheus-metrics. #4686
- Decrease logging frequency for no local transaction status tablet from on every warning to once per second. #5035
- Fix MonoTime subtraction implementation to return
result
of the operation instead oflhs
. #5020 - When dropping tables, delete snapshot directories for deleted tables. #4756
- Set up a global leader balance threshold while allowing progress across tables. Add
load_balancer_max_concurrent_moves_per_table
andload_balancer_max_concurrent_moves
to improve performance of leader moves. And properly update state for leader stepdowns to prevent check failures. #5021 #5181 - Reduce default value of
load_balancer_max_concurrent_moves
from10
to2
. #5461 - Improve bootstrap logic for resolving transaction statuses. #5215
- Avoid duplicate DNS lookup requests while handling
system.partitions
table requests. #5225 - Handle write operation failures during tablet bootstrap. #5224
- Ensure
strict_capacity_limit
is set when SetCapacity is called to prevent ASAN failures. #5222 - Drop index upon failed backfill. For YCQL, generate error to CREATE INDEX call if index backfill fails before the call is completed. #5144 #5161
- Allow overflow of single-touch cache if multi-touch cache is not consuming space. #4495
- Don't flush RocksDB before restoration. #5184
- Fix
yugabyted
fails to start UI due to class binding failure. #5069 - On
yb-tserver
restart, prioritize bootstrapping transaction status tablets. #4926 - For
yb-admin
commands, clarify syntax for namespace. #5482 - Improve for-loops to avoid unnecessary copies and remove range-loop-analysis warnings. #5458
- Apply large transaction in a separate thread if the transaction will not fit into the limit of key-value pairs. The running transaction is removed from memory after all its intents are applied and removed. #1923
- Fix
SEGV
in Master UI when registering YB-TServer from Raft. #5501 - Heartbeats process only new consensus information unless
master_ignore_stale_cstate
flag is disabled. #5301 - Remove KernelStackWatchdog and use LongOperationTracker to waste less resources per thread. #5226
- Change the default value of
metric_node_name
flag fromDEFAULT_NODE_NAME
tohostname:port
. #4859
Yugabyte Platform
- For S3 backups, install
s3cmd
required for encrypted backup and restore flows. #5593 - When creating on-premises provider, remove
YB_HOME_DIR
if not set. #5592 - Update to use templatized values for setting TLS flags in Helm Charts. #5424
- For YCQL client connectivity, allow downloading root certificate and key from Certificates menu. #4957
- Bypass CSRF checks when registering and logging in due to bug with CSRF token not being set. #5533
- Set node certificate field to subject of CA instead of issuer of CA. #5377
- Add option in universe creation form to control whether node_exporter is installed on provisioned nodes (including creating prometheus user + group). #5421
- Add Microsoft Azure provider UI and related changes to Create Universe form. #5378
- Create default network resources for multi-region deployments in Microsoft Azure if they don't exist and store for use during universe creation. #5388
- For Microsoft Azure, use preexisting network resources. #5389
- Set Assign Public IP correctly using the UI. #5463
- Fix
AreLeadersOnPreferredOnly
times out when setting preferred zones in Edit Universe. #5406 - Fix
yb_backup.py
script to removesudo
requirement and correctly change user. #5440 - Add CSRF tokens to forms. #5419
- Allow configuration of OIDC scope (for SSO use) as well as email attribute field. #5465
- If Username and Password are left empty in the Custom SMTP Configuration, remove
smtpUsername
andsmtpPassword
from the payload. #5439 - Add the UTC label to indicate
cron
expression must be relative to UTC. Also, add help text indicating when next scheduled job will run. #4709 - Move the source of truth for communication ports to the universe level. #5353
- Use appropriate range for time period (over entire step window rather than last minute) when querying Prometheus for metrics. #4203
- Support backing up encrypted at rest universes. #3118
- Only display "Rolling Upgrade Delay" if upgrade type is "Rolling". #5362
- Add guarding against never-ending loop when running the
yb_backup.py
script and yb-server doesn't have any data directories. #5358 - Fix metric panel displaying incorrect time interval when reloading after changing graph filter. #5294
- Fix logging on health check script failure. #5340
- Rename the label for enabling server-side encryption in S3 backups to Encrypt Backup for both backups and scheduled backup modals. #5273
- Add a VM backup script that includes Prometheus backups. #5120
- Add
BackupAdmin
role to authorize users to create backup and restore tasks on universes, but not given other administration privileges. #4694 - Modify the
yb_backup.py
script to add "round-robin" behavior to thread allocations to better spread loads across nodes in parallel during snapshot upload or download. #4987 - Fix the invalid
yb_backup.py
script behavior if--keyspace
flag is not specified. #5199 - Add subtasks to check number of pods deployed during upgrades. #5174
- Make use of parallelism argument for
yb_backup.py
script. Explicitly pass the number of threads to concurrently run backup script through UI. The form defaults to the backup script default value of 8 threads if no value is provided. #5283 - The Remove Node action should be available if the Add Node action fails. #5123
- Add ability for user to specify the S3 backup target host that their bucket is located on. #3684
- Check if on-premise node instance has a universe that is using it and make a link to the Universe Overview page. #4758
- Fix
Cannot read property 'displayName' of undefined
error. #5267 - Use Host Names setting is not displaying correctly in Provider Configuration page. #5252
- Disable glob before running the cleanup of old log files. #5169
- Schedule job for
setUniverseKey
should handle errors on per-customer basis. #3142 - After Yugabyte Platform (YugaWare) restart, any in-progress tasks should be cancelled and marked as failed. #5045
- Redirect from login back to index if session is already validated. Logout should redirect to the login endpoint. #5019
- Fix updating of Universes list after creating new universe. #4784
- Fix restore payload when renaming table to include keyspace. And check for keyspace if tableName is defined and return invalid request code when keyspace is missing. #5178
- Add the ability to override communication port default values during universe creation. #5354
v2.3.1.0
Released: September 15, 2020 (2.3.1.0-b15).
Downloads
Binaries
Docker
docker pull yugabytedb/yugabyte:2.3.1.0-b15
Features and changes
YSQL
- Fixes OOM when using
COPY <table> FROM <file>
to load data from a large file to a table. #5453 - Enable 2DC replication bootstrap for YSQL tables. This allows the producer to know where to start replicating from. #5601
- Fix
CREATE TABLE
is 4-5x slower using Docker on Mac than not using Docker. Speeds up table creation by buffering writes to postgres system tables, caching pinned objects, and significantly reducing write RPC calls. #3503 - Roll back the catalog version made in commit
46f3701
so that 2.3 upgrades can proceed. #5408
Core database
- Quickly evict known unresponsive tablet servers from the tablet location cache. Applies only to follower reads. For example, when tablet servers are unresponsive or dead — not sending heartbeats to the master for 5 minutes. This could also happen after decommissioning nodes. #1052
Yugabyte Platform
- Add search input and data sorting to the on-premises instances table list. Click arrows next to column titles to sort. Use the Search form to search multiple columns. #4757
- Add the ability to change the user role (
Admin
,ReadOnly
, orBackupAdmin
) from the UI by an admin. Also, fix stale users list after creation or deletion of a user and disable Save buttons at Customer Profile tabs forReadOnly
users. #5311 - Before deleting a universe, stop the master/tserver processes and release the instance so that if started again, the processes are not still running. #4953
v2.3.2.0
Released: September 29, 2020 (2.3.2.0-b37).
Downloads
Binaries
Docker
docker pull yugabytedb/yugabyte:2.3.2.0-b37
Features and changes
YSQL
- Enable
pg_stat_statements
extension by default. #5750 - Clean up unused
PgStatement
objects on hot path after usage to avoid memory consumption. #5374 - Fix restarting writes (
INSERT
andUPDATE)
on transaction conflicts and avoid surfacing transaction errors to user applications. #4291 - Enable batching of updates for YSQL procedures and functions. #5257
YCQL
- Add additional system queries (
SELECT_LOCAL
,SELECT_LOCAL_TOKENS
, andSELECT_PEERS_TOKENS
) for caching. #5753 - Fix crash for YCQL index scan when
ORDER BY
columns exist in table, but not in the chosen index, and raise an error. #5690
Core database
- Allow setting a custom replacement policy per table through
yb_admin
. #5368 - Do not clean up transaction that potentially was committed. #5789 #5783 #5755
- Fix CDC shutdown races. Moved thread pool shutdown before other CDC objects cleanup. Added explicit CDC poller clear in CDC consumer shutdown. #5583
- Fix preparer deadlock after switching from leader to follower. #5741
- Fix
RaftGroupMetadata
construction issues found using PVS Studio. #5578 - Change
rlimit
flags to have at-least semantics. Only change the process soft limit if it is smaller than the limit specified by the corresponding flag value. #5715 - Add extra logging when initializing
syscatalog
. #5708 - Set default
webserver_interface
torpc_bind_address
IP/host component. #5320 - Add environmental variables functionality to
yugabyted
. New environment variables for YSQL (YSQL_USER
,YSQL_PASSWORD
, andYSQL_DB
) and YCQL(YCQL_USER
,YCQL_PASSWORD
, andYCQL_KEYSPACE
). (#5499 - Improve cleanup of aborted transactions. #5669
- Fix dumping an empty batch by updating MaxReplicateOpId. #5679
- Set default staleness for read from followers (using
max_stale_read_time_ms
) to 60 seconds. Previously, a follower could return requested data no matter how state the data is. #5590 - Calculate the placement UUID from Raft upon tserver registration. #5558
- Enable ignoring deleted tablets. #5122
- Remove unreachable nodes from UI (
/tablet-servers
page) after nodes are removed from the cluster. #4759
Yugabyte Platform
- Fix issue parsing multiple comma-separated addresses for
rpc_bind_addresses
. #5634 - Add functionality to delete old backups after a specified time period. In the Create Backup dialog, added Number of Days to Retain Backup field. #4493
- Combine the Platform and community Grafana dashboards. Grafana dashboard shows the node status and other panels correctly. #5707
- Sum table sizes across all nodes of a universe. #5745
- Fixes releasing OnPrem nodes should not delete them from the universe metadata. Fixes releasing a node should only attempt to shut down processes on that node. Nodes can be re-added using Node > Actions > Add flow. #5703 #5791 #5792
- Add additional property to MetricsPanel to handle width of Replication graph and increase height. Aggregates replication metrics values from all tservers into an average and max value graph across all tservers. #5429 #5613 #5632
- Update yb-client JAR version to
0.8.1
. #5759 - Stop and restart relevant services during Yugabyte Platform backup. Backups stop and restart services. Restores stop and restart Yugabyte Platform and Prometheus, if necessary. #5685
- Authenticate API tokens even when using OAuth. This allows
curl
requests. #5587 - Fix Metrics tab and columns misalignment on table page. #5582
- Rename In Use column to Universe Name and show
NOT USED
if the node is not in use. #5713 - Do not generate node certificates for custom certificates. #5545
- Make instance name field optional for on-premises provider. #5234
- Fix button for Metrics tab to reset active and focus state when key is unpressed. #5620
- Add Docker- and VM-aware options to the backup script and will include Prometheus snapshots. #5120 #5615 #5612 #5614
v2.3.3.0
Released: October 26, 2020 (2.3.3.0-b106).
Downloads
Binaries
Docker
docker pull yugabytedb/yugabyte:2.3.3.0-b106
New features
YSQL
- Load data from a large file to a table using the improved
COPY <table> FROM <file>
statement. For details, seeCOPY
. Improved memory management should prevent out-of-memory (OOM) issues. - Specify smaller transaction sizes using the
COPY OPTION
ROWS_PER_TRANSACTION
. For an example, see Import a large table using smaller transactions.
Yugabyte Platform
- Introducing beta support for Azure IaaS orchestration in Yugabyte Platform cloud provider configuration. (6094, 6020)
- Yugabyte Platform operations now allow promoting a Yugabyte TServer only node to run YB-Master and YB-TServer process. (5831)
Core database
- [YSQL] Introducing support for Event Triggers that are global to a particular database and are capable of capturing DDL events. (2379)
Improvements
Yugabyte Platform
- Enhancements to on-premises deployment workflows:
- Do not fail universe creation if cronjobs can't be created for on-premises. (5939)
- Remove pre-provision script requirement for airgapped installations. (5929)
- "Sudo passwordless" in on-premises cloud provider configuration toggle is renamed.
- Allow
yugabyte
user to belong to other user groups in Linux. (5943) - Added a new "Advanced" section in on-prem cloud provider configuration which includes:
- Use hostnames
- Desired home directory
- Node exporter settings
- Improvements to installation of Prometheus Node Exporter utility workflow. (5926)
- The node exporter option is now available in the cloud configuration under advanced settings.
- Supports bringing your own node exporter user.
- Make YEDIS API optional for new Universes and no change in behavior of existing universes. (5207)
- UI/UX improvements for YB Platform:
- For YugabyteDB universes with replication factor (RF) > 3, change the default
min_num
replicas for even distribution of AZs across all regions. (5426) - Added functionality to create IPv6-enabled universe in Kubernetes. (5309, 5235)
Core database
- Support for SQL/JSON Path Language (
jsonb_path_query
). (5408) - Remove range-loop-analysis and fallthrough errors in our code. (5812)
- Incorrect index update if used expression result is still unchanged. (5898)
- Various improvements to
RaftConsensus
tests. (4548, 5912) - As part of the tablet splitting feature:
- Improve performance for sequences by using higher cache value by default. Controlled by a YB-TServer configuration flag
--ysql_sequence_cache_minval
. (6041) - Added compatibility mode in the
yb_backup.py
script for YugabyteDB version earlier than v2.1.4. (5810) - Stability improvements to core database - make exponential backoff on lagging RAFT followers send NOOP. (5527)
- Added use of separate metrics objects for
RegularDB
andIntentsDB
. (5640) - Improvements to 2DC (xCluster replication) by adding TLS support for cleaning up
cdc_state
table. (5905)
Bug fixes
Yugabyte Platform
- Fix for Universe disk usage shows up empty on the universe page. (5548)
- Fix on on-premises backup failures due to file owned by the root user. (6062)
- Fix for a bug where user operation to perform a change to nodes count by AZ was doing a full move. (5335)
- Fixes for Yugabyte Platform data backup script for Replicated-based installations
- Fixes to Client Certificate start time to use UTC during download. (6118)
- Fixes for migration if no access keys exist yet. (6099)
- Fix to resolve issues caused by starting a YB-TServer node when another YB-Master node is down in the universe. (5739)
- Use the correct disk mount point while calculating disk usage of logs. (5983)
- Fixes to delete backups for TLS-enabled universes. (5980)
Core database
- Fix for bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. (5811)
- Improve fault tolerance of DDLs and reduce version mismatch errors in YSQL. (3979, 4360)
- Fixes to incorrect column-ids in the restored table if the original table was altered. (5958)
- Fixes timeout bug in Yugabyte Platform when there are read replicas. This fix will ignore read replica YB-TServers when running
AreLeadersOnPreferredOnly
. (6081) - Fixes to restore of YSQL Backups after dropping and recreating a database. (5651)
- Fixes to a bug with the duplicate row detection that allows a unique index to get created when the table is not unique on the index columns. This is fixed by setting the read time for backfill write requests to be the safe time chosen for backfill read requests rather than the hard-coded time
50
. (5811)