Skip to content

Quota Management

Celeborn offers administrators flexibility by allowing them to set quotas for individual users and providing a system-level default quota for users without a specific named quota. This feature ensures control and customization in managing system quotas.

When celeborn.quota.enabled is set to true, the Master enforces quota limits using the QuotaManager. Similarly, if this setting is enabled on the client side, the LifecycleManager will ask the Master to verify whether the current user has sufficient quota before shuffle registration. Should there be insufficient quota, the LifecycleManager will revert to using Spark's default shuffle service.

Quota Indicators

Celeborn supports fine-grained quota management, including four indicators:

  • celeborn.quota.tenant.diskBytesWritten: Maximum allowed size of disk write files, of which default value Long.MAX_VALUE.
  • celeborn.quota.tenant.diskFileCount: Maximum allowed number of disk write files, of which default value is Long.MAX_VALUE.
  • celeborn.quota.tenant.hdfsBytesWritten: Maximum allowed size of HDFS write files, of which default value Long.MAX_VALUE.
  • celeborn.quota.tenant.hdfsFileCount: Maximum allowed number of HDFS write files, of which default value is Long.MAX_VALUE.

User Identifier

The LifecycleManager will request the Master to check the quota for the current user defined by user setting. Users can set celeborn.quota.identity.provider to choose an identity provider. Celeborn support the following types at present:

  • org.apache.celeborn.common.identity.HadoopBasedIdentityProvider: The username will be obtained by UserGroupInformation.getUserName(), tenant id will be default.
  • org.apache.celeborn.common.identity.DefaultIdentityProvider: The username and tenant id are default values or user-specific values set by celeborn.quota.identity.user-specific.tenant and celeborn.quota.identity.user-specific.userName.

By default, Celeborn uses org.apache.celeborn.common.identity.DefaultIdentityProvider. Users can also implement their own identity provider by inheriting the org.apache.celeborn.common.identity.IdentityProvider interface.

QuotaManager

QuotaManager supports to check whether quota is available and manage quota configurations for Master. QuotaManager uses the dynamic config service to store quota settings. For example, there are some quota configurations as follows:

The quota for user tenant_01.Jerry is

  • diskBytesWritten: 100G
  • diskFileCount: 10000
  • hdfsBytesWritten: 10G
  • diskFileCount: Long.MAX_VALUE

The quota for tenant id tenant_01 is

  • diskBytesWritten: 10G
  • diskFileCount: 1000
  • hdfsBytesWritten: 10G
  • diskFileCount: Long.MAX_VALUE

The quota for system default is

  • diskBytesWritten: 1G
  • diskFileCount: 100
  • hdfsBytesWritten: 1G
  • diskFileCount: Long.MAX_VALUE

FileSystem Store Backend

This backend reads quota settings from a user-specified dynamic config file. For more information on using the database store backend, refer to filesystem config service. Here's an example quota setting YAML file of above quota examples:

-  level: SYSTEM
   config:
     celeborn.quota.tenant.diskBytesWritten: 1G
     celeborn.quota.tenant.diskFileCount: 100
     celeborn.quota.tenant.hdfsBytesWritten: 1G

-  tenantId: tenant_01
   level: TENANT
   config:
     celeborn.quota.tenant.diskBytesWritten: 10G
     celeborn.quota.tenant.diskFileCount: 1000
     celeborn.quota.tenant.hdfsBytesWritten: 10G
   users:
     - name: Jerry
       config:
         celeborn.quota.tenant.diskBytesWritten: 100G
         celeborn.quota.tenant.diskFileCount: 10000

Database Store Backend

This backend reads quota settings from a user-specified database. For more information on using the database store backend, refer to database config service. Here's an example quota setting sql of above quota examples: 

# SYSTEM level configuration
INSERT INTO `celeborn_cluster_system_config` ( `id`, `cluster_id`, `config_key`, `config_value`, `type`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 1, 'celeborn.quota.tenant.diskBytesWritten', '1G', 'QUOTA', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 2, 1, 'celeborn.quota.tenant.diskFileCount', '100', 'QUOTA', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 3, 1, 'celeborn.quota.tenant.hdfsBytesWritten', '1G', 'QUOTA', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );

# TENANT/TENANT_USER level configuration
INSERT INTO `celeborn_cluster_tenant_config` ( `id`, `cluster_id`, `tenant_id`, `level`, `name`, `config_key`, `config_value`, `type`, `gmt_create`, `gmt_modify` )
VALUES
    ( 1, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.diskBytesWritten', '10G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 2, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.diskFileCount', '1000', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 3, 1, 'tenant_01', 'TENANT', '', 'celeborn.quota.tenant.hdfsBytesWritten', '10G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 4, 1, 'tenant_01', 'TENANT_USER', 'Jerry', 'celeborn.quota.tenant.diskBytesWritten', '100G', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' ),
    ( 5, 1, 'tenant_01', 'TENANT_USER', 'Jerry', 'celeborn.quota.tenant.diskFileCount', '10000', 'master', '2024-02-27 22:08:30', '2024-02-27 22:08:30' );