querying-aws-redshift
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseQuery AWS Redshift System Tables
查询AWS Redshift系统表
Overview
概述
Works best with the AWS MCP server for sandboxed execution and audit logging. All commands below use the AWS CLI and work in any environment with configured AWS credentials. Use IAM roles or temporary credentials; avoid long-lived access keys.
Redshift can publish system tables — the monitoring data such as , , and — to S3 Tables as continuously-updated Apache Iceberg tables.
SYS_*sys_query_historysys_query_detailsys_connection_logTerminology used throughout: system table refers to a dataset generally, and each one maps 1:1 to a published Iceberg table. Where this skill says view, it means specifically the live in-cluster object you query on the cluster itself — that is a view, and it is a different thing from the published S3 Tables copy. This applies to both Provisioned clusters and Serverless namespaces. It is an opt-in extension of the existing logging APIs. Published tables are read-only, stored in the AWS-managed table bucket, and queryable via any Iceberg-compatible engine including Amazon Athena and Amazon Redshift itself.
SYS_*SYS_aws-redshiftQuerying the S3 Tables copy is preferred over the live in-cluster views when analyzing historical or high-volume system-table data because:
SYS_- The in-cluster views have a limited retention window; S3 Tables retains history well beyond it.
SYS_ - Querying S3 Tables adds no load to the running Redshift cluster.
- The logs are Iceberg tables, so they can be queried at scale from any Iceberg-compatible engine and joined with other lake data.
此功能与AWS MCP服务器配合使用效果最佳,可实现沙箱执行和审计日志记录。以下所有命令均使用AWS CLI,可在任何配置了AWS凭证的环境中运行。建议使用IAM角色或临时凭证,避免使用长期访问密钥。
Redshift可将系统表(即监控数据,如、和)以持续更新的Apache Iceberg表形式发布到S3 Tables。
SYS_*sys_query_historysys_query_detailsys_connection_log全文使用的术语说明:系统表泛指数据集,每个系统表与一个已发布的Iceberg表一一对应。当本技能提及**视图时,特指你在集群上直接查询的集群内实时对象——它是一个视图,与已发布的S3 Tables副本并非同一对象。此功能适用于预置集群和无服务器命名空间**,是现有日志API的可选扩展。已发布的表为只读状态,存储在AWS托管的表存储桶中,可通过Amazon Athena、Amazon Redshift等任何兼容Iceberg的引擎进行查询。
SYS_*SYS_aws-redshift在分析历史或高容量系统表数据时,查询S3 Tables副本比查询集群内实时视图更具优势,原因如下:
SYS_- 集群内视图的保留窗口有限;而S3 Tables的历史数据保留时间远超出该窗口。
SYS_ - 查询S3 Tables不会对运行中的Redshift集群造成任何负载。
- 日志以Iceberg表形式存储,可通过任何兼容Iceberg的引擎进行大规模查询,还能与其他湖仓数据关联。
Decision Tree
决策树
| User intent | Use this skill? | Alternative |
|---|---|---|
| Turn on S3 Tables log publishing for a cluster or namespace | Yes | — |
| Confirm a cluster/namespace is publishing / find its S3 Tables namespace | Yes | — |
| Querying non-realtime data from Redshift system tables | Yes | — |
| Build daily/weekly/monthly dashboard for Redshift monitoring and auditing | Yes | — |
| Selectively stop S3 Tables publishing | Yes | — |
| Query published system tables from Redshift (cross-database) | Yes | — |
| Query published system tables from Athena | Yes | — |
Inspect the current, real-time | No | Query the |
| Query data inside customer tables | No | Direct Redshift SQL on the cluster |
| 用户意图 | 是否使用本技能 | 替代方案 |
|---|---|---|
| 为集群或命名空间开启S3 Tables日志发布 | 是 | — |
| 确认集群/命名空间是否正在发布日志 / 查找其S3 Tables命名空间 | 是 | — |
| 查询Redshift系统表的非实时数据 | 是 | — |
| 构建Redshift监控和审计的每日/每周/每月仪表板 | 是 | — |
| 选择性停止S3 Tables日志发布 | 是 | — |
| 从Redshift(跨数据库)查询已发布的系统表 | 是 | — |
| 从Athena查询已发布的系统表 | 是 | — |
检查集群上当前实时的 | 否 | 直接在集群上查询 |
| 查询用户表内的数据 | 否 | 在集群上直接执行Redshift SQL |
Supported Data Sources
支持的数据源
| Compute type | Enable / disable API | Status API | Granularity options |
|---|---|---|---|
| Redshift Provisioned cluster | | | |
| Redshift Serverless namespace | | | |
Both compute types publish into the same AWS-managed table bucket and are queried identically once published. They differ only in the enable/disable API surface and in the casing of the status response — see the flag and field tables in Common Tasks.
aws-redshiftNot covered by this skill: Redshift audit logs delivered to S3 or CloudWatch (, , ), which use the separate mechanism on Serverless and are not system tables.
useractivityloguserlogconnectionlog--log-exportsSYS_*| 计算类型 | 启用/禁用API | 状态API | 粒度选项 |
|---|---|---|---|
| Redshift预置集群 | | | |
| Redshift无服务器命名空间 | 带 | | |
两种计算类型都会将数据发布到同一个AWS托管的表存储桶,发布完成后的查询方式完全相同。二者仅在启用/禁用API接口以及状态响应的字段格式上存在差异——详情请参见常见任务中的标志和字段表。
aws-redshift本技能不涵盖以下内容:交付到S3或CloudWatch的Redshift审计日志(、、),这些日志使用无服务器环境中独立的机制,不属于系统表范畴。
useractivityloguserlogconnectionlog--log-exportsSYS_*Common Tasks
常见任务
1. Check If Configured
1. 检查配置状态
Before querying, confirm the cluster or namespace is publishing to S3 Tables.
bash
undefined查询前,请确认集群或命名空间是否已配置为向S3 Tables发布日志。
bash
undefinedProvisioned
预置集群
aws redshift describe-logging-status --region <REGION> --cluster-identifier <CLUSTER_ID>
aws redshift describe-logging-status --region <REGION> --cluster-identifier <CLUSTER_ID>
Serverless
无服务器命名空间
aws redshift-serverless get-namespace --region <REGION> --namespace-name <NAMESPACE_NAME>
**Interpret the response.** The two compute types return the *same* information under **different field names and casing** — Provisioned uses PascalCase under `S3Tables`, Serverless uses camelCase under `namespace.s3TablePublishStatus`:
| Meaning | Provisioned (`describe-logging-status`) | Serverless (`get-namespace`) |
|---|---|---|
| Not enabled | `LoggingEnabled: false` or no `S3Tables` block | no `s3TablePublishStatus` block |
| Destination includes S3 Tables | `LogDestinationType` contains `s3table` | `logDestinationType` contains `s3table` |
| List of published `SYS_*` tables | `S3Tables.S3Tables` | `namespace.s3TablePublishStatus.s3Tables` |
| **The exact S3 Tables namespace** (required for querying) | `S3Tables.S3TableNamespace` | `namespace.s3TablePublishStatus.s3TableNamespace` |
| Granularity | `S3Tables.S3TableGranularity` (`cluster`/`account`) | `namespace.s3TablePublishStatus.s3TableGranularity` (`namespace`/`account`) |
| Per-table last ingest time | `S3Tables.LastIngestionTimes` | `namespace.s3TablePublishStatus.lastIngestionTimes` |
| All available system tables published | `S3Tables.EnabledAll` | `namespace.s3TablePublishStatus.enabledAll` |
Notes:
- `LogDestinationType` is a **comma-joined list** when more than one destination is active — e.g. `"cloudwatch,s3table"`. Test with a substring/contains check, not equality against `s3table`.
- An empty `LastIngestionTimes` / `lastIngestionTimes` map, or a table listed as published but absent from the map, means data for that table may still be in flight. Compare successive values to confirm new data is landing.
- On Serverless, do **not** read the top-level `logExports` field for this feature — that field carries the CloudWatch/S3 audit logs (`useractivitylog`, `userlog`, `connectionlog`) and is unrelated to `SYS_*` S3 Tables publishing.aws redshift-serverless get-namespace --region <REGION> --namespace-name <NAMESPACE_NAME>
**解读响应结果**。两种计算类型返回的是**相同信息**,但字段名称和格式不同——预置集群在`S3Tables`下使用 PascalCase,无服务器命名空间在`namespace.s3TablePublishStatus`下使用 camelCase:
| 含义 | 预置集群(`describe-logging-status`) | 无服务器命名空间(`get-namespace`) |
|---|---|---|
| 未启用 | `LoggingEnabled: false` 或无`S3Tables`块 | 无`s3TablePublishStatus`块 |
| 目标包含S3 Tables | `LogDestinationType`包含`s3table` | `logDestinationType`包含`s3table` |
| 已发布的`SYS_*`表列表 | `S3Tables.S3Tables` | `namespace.s3TablePublishStatus.s3Tables` |
| **准确的S3 Tables命名空间**(查询必需) | `S3Tables.S3TableNamespace` | `namespace.s3TablePublishStatus.s3TableNamespace` |
| 粒度 | `S3Tables.S3TableGranularity`(`cluster`/`account`) | `namespace.s3TablePublishStatus.s3TableGranularity`(`namespace`/`account`) |
| 各表的最后摄取时间 | `S3Tables.LastIngestionTimes` | `namespace.s3TablePublishStatus.lastIngestionTimes` |
| 已发布所有可用系统表 | `S3Tables.EnabledAll` | `namespace.s3TablePublishStatus.enabledAll` |
注意事项:
- 当存在多个目标时,`LogDestinationType`是**逗号分隔的列表**——例如`"cloudwatch,s3table"`。请使用子字符串/包含性检查,而非直接判断是否等于`s3table`。
- `LastIngestionTimes` / `lastIngestionTimes`为空,或某张表已列在已发布列表中但未出现在该映射里,意味着该表的数据可能仍在传输中。可对比连续多次查询的结果,确认新数据是否已落地。
- 在无服务器环境中,请勿通过顶层`logExports`字段查看此功能——该字段承载的是CloudWatch/S3审计日志(`useractivitylog`、`userlog`、`connectionlog`),与`SYS_*` S3 Tables发布无关。2. Enable (if not configured)
2. 启用日志发布(若未配置)
bash
undefinedbash
undefinedProvisioned
预置集群
aws redshift enable-logging --region <REGION> --cluster-identifier <CLUSTER_ID> --log-destination-type s3table --log-exports <SYS_TABLE>... --s3-table-granularity <cluster|account> --s3-table-kms-key-id <KMS_KEY_ARN>
aws redshift enable-logging --region <REGION> --cluster-identifier <CLUSTER_ID> --log-destination-type s3table --log-exports <SYS_TABLE>... --s3-table-granularity <cluster|account> --s3-table-kms-key-id <KMS_KEY_ARN>
Serverless
无服务器命名空间
aws redshift-serverless update-namespace --region <REGION> --namespace-name <NAMESPACE_NAME> --log-destination-type s3table --s3-table-names <SYS_TABLE>... --s3-table-action Enable --s3-table-granularity <namespace|account> --s3-table-kms-key-id <KMS_KEY_ARN>
`--s3-table-kms-key-id` is part of both commands deliberately, not an optional add-on. Omitting it does not fail — the tables fall back to an AWS-owned key you cannot audit, restrict by policy, or revoke. Because `SYS_*` tables carry `query_text`, `user_name`, and `remote_host`, treat the customer-managed key as the default and drop the flag only for throwaway environments.
Enable from AWS console Amazon Redshift Console > Clusters > select your cluster > Tabs > Integrations / System table integration
**The two compute types take different flags for the same feature.** Do not carry Provisioned flag names over to Serverless:
| Purpose | Provisioned (`enable-logging`) | Serverless (`update-namespace`) |
|---|---|---|
| Which system tables to publish | `--log-exports` | `--s3-table-names` |
| Enable vs disable | separate `enable-logging` / `disable-logging` operations | `--s3-table-action Enable` \| `Disable` |
| Granularity | `--s3-table-granularity` `cluster` \| `account` | `--s3-table-granularity` `namespace` \| `account` |
| Customer-managed KMS key | `--s3-table-kms-key-id` | `--s3-table-kms-key-id` |
| Validate without applying | `--dry-run` | `--dry-run` |
Notes:
- Granularity: Provisioned supports `cluster` (default) or `account`; Serverless supports `namespace` (default) or `account`.
- `cluster`/`namespace` granularity → one S3 table per cluster/namespace; `account` → one shared table for all clusters/namespaces per account per region.
- Use `all` to publish all available `SYS_*` tables — `--log-exports all` on Provisioned, `--s3-table-names all` on Serverless.
- **Encryption at rest is strongly recommended for production.** Without `--s3-table-kms-key-id` the published tables are encrypted with an AWS-owned key, which you cannot audit, restrict by policy, or revoke. `SYS_*` tables carry `query_text`, `user_name`, and `remote_host` (see [Security Considerations](#security-considerations)), so pass a customer-managed key. Grant key access using the complete key policy in `${SKILL_DIR}/references/security.md` rather than an abbreviated action list — it needs **two** service principals (`systemtables.redshift.amazonaws.com` for publishing and `maintenance.s3tables.amazonaws.com` for table maintenance/compaction). Provisioning only the publishing principal lets writes succeed while compaction silently fails.
- Both operations accept `--dry-run` to validate the request without changing anything. Provisioned returns a `DryRunOperation` error on success ("Request would have succeeded, but DryRun flag is set"); Serverless returns an empty body and exit code 0. Note that the Serverless dry-run validates request *shape* only, not parameter values, so a successful dry-run there does not guarantee the values are accepted.
**Disable selectively:**
```bashaws redshift-serverless update-namespace --region <REGION> --namespace-name <NAMESPACE_NAME> --log-destination-type s3table --s3-table-names <SYS_TABLE>... --s3-table-action Enable --s3-table-granularity <namespace|account> --s3-table-kms-key-id <KMS_KEY_ARN>
`--s3-table-kms-key-id`是两个命令的必填参数,而非可选附加项。省略该参数不会导致命令失败——表会默认使用AWS托管密钥,但你无法对其进行审计、策略限制或撤销操作。由于`SYS_*`表包含`query_text`、`user_name`和`remote_host`等敏感信息,建议默认使用客户托管密钥,仅在临时测试环境中可省略该参数。
也可通过AWS控制台启用:Amazon Redshift控制台 > 集群 > 选择你的集群 > 标签页 > 集成 / 系统表集成
**两种计算类型针对同一功能使用不同的命令参数**。请勿将预置集群的参数名称直接用于无服务器环境:
| 用途 | 预置集群(`enable-logging`) | 无服务器命名空间(`update-namespace`) |
|---|---|---|
| 指定要发布的系统表 | `--log-exports` | `--s3-table-names` |
| 启用/禁用 | 分别使用`enable-logging` / `disable-logging`操作 | `--s3-table-action Enable` \| `Disable` |
| 粒度 | `--s3-table-granularity` `cluster` \| `account` | `--s3-table-granularity` `namespace` \| `account` |
| 客户托管KMS密钥 | `--s3-table-kms-key-id` | `--s3-table-kms-key-id` |
| 验证请求而不执行 | `--dry-run` | `--dry-run` |
注意事项:
- 粒度:预置集群支持`cluster`(默认)或`account`;无服务器命名空间支持`namespace`(默认)或`account`。
- `cluster`/`namespace`粒度 → 每个集群/命名空间对应一个S3表;`account`粒度 → 每个区域内的所有集群/命名空间共享一个表。
- 使用`all`可发布所有可用的`SYS_*`表——预置集群使用`--log-exports all`,无服务器命名空间使用`--s3-table-names all`。
- **生产环境强烈建议启用静态加密**。若未指定`--s3-table-kms-key-id`,已发布的表将使用AWS托管密钥加密,你无法对其进行审计、策略限制或撤销操作。`SYS_*`表包含`query_text`、`user_name`和`remote_host`等敏感信息(详见[安全注意事项](#security-considerations)),因此请传入客户托管密钥。请使用`${SKILL_DIR}/references/security.md`中的完整密钥策略授予密钥访问权限,而非仅使用简化的操作列表——该策略需要**两个**服务主体(`systemtables.redshift.amazonaws.com`用于发布,`maintenance.s3tables.amazonaws.com`用于表维护/压缩)。若仅配置发布主体,写入操作可成功执行,但压缩操作会静默失败。
- 两个命令均支持`--dry-run`参数,可验证请求的有效性而不实际更改任何配置。预置集群在验证成功时会返回`DryRunOperation`错误("Request would have succeeded, but DryRun flag is set");无服务器命名空间会返回空响应体且退出码为0。请注意,无服务器环境的dry-run仅验证请求的格式,不验证参数值,因此dry-run成功并不代表参数值会被接受。
**选择性禁用日志发布:**
```bashProvisioned
预置集群
aws redshift disable-logging --region <REGION> --cluster-identifier <CLUSTER_ID> --log-destination-type s3table --log-exports <SYS_TABLE>...
aws redshift disable-logging --region <REGION> --cluster-identifier <CLUSTER_ID> --log-destination-type s3table --log-exports <SYS_TABLE>...
Serverless
无服务器命名空间
aws redshift-serverless update-namespace --region <REGION> --namespace-name <NAMESPACE_NAME> --log-destination-type s3table --s3-table-names <SYS_TABLE>... --s3-table-action Disable
undefinedaws redshift-serverless update-namespace --region <REGION> --namespace-name <NAMESPACE_NAME> --log-destination-type s3table --s3-table-names <SYS_TABLE>... --s3-table-action Disable
undefined3. Verify Permissions
3. 验证权限
Full setup commands for both paths: . Load it before creating roles or registering resources.
${SKILL_DIR}/references/permissions-setup.mdAthena path — needs the catalog registered in Glue, a workgroup with an output location, and S3 Tables read permissions. Confirm the catalog is queryable:
s3tablescatalog/aws-redshiftbash
aws glue get-databases --region <REGION> \
--catalog-id "<ACCOUNT>:s3tablescatalog/aws-redshift"Namespaces returned → registered and queryable. / → the S3 Tables integration is not enabled (S3 console > Table buckets > Enable integration). Encrypt the workgroup output location — Athena writes full result sets, including and , to S3.
EntityNotFoundExceptionCATALOG_NOT_FOUNDquery_textuser_nameRedshift auto-mount path — needs a Provisioned RA3 cluster and a four-step setup: create the role (trust policy must name both and , the latter with all four of , , , ), attach it to the cluster, register the table bucket with Lake Formation, and add the Redshift service-linked roles to . Constraints that cause most failures:
query_s3_tablesredshift.amazonaws.comlakeformation.amazonaws.comsts:AssumeRolests:SetContextsts:SetSourceIdentitysts:TagSessionReadOnlyAdmins- Condition both trust statements on — a bare service principal is a confused-deputy risk.
aws:SourceAccount - Do not attach to the cluster's query role. It is needed only by the principal performing setup, and only during setup. The cluster's role needs read access alone.
AWSLakeFormationDataAdmin - Auto-mount is a poll, not a callback — the catalog can take up to 300 seconds to appear in . A cluster reboot forces immediate discovery.
pg_database
两种路径的完整设置命令:。创建角色或注册资源前,请先查看该文件。
${SKILL_DIR}/references/permissions-setup.mdAthena路径——需要在Glue中注册目录、配置带有输出位置的工作组,以及S3 Tables读取权限。确认目录可查询:
s3tablescatalog/aws-redshiftbash
aws glue get-databases --region <REGION> \
--catalog-id "<ACCOUNT>:s3tablescatalog/aws-redshift"返回命名空间 → 目录已注册且可查询。返回 / → S3 Tables集成未启用(S3控制台 > 表存储桶 > 启用集成)。加密工作组输出位置——Athena会将完整结果集(包括和)写入S3。
EntityNotFoundExceptionCATALOG_NOT_FOUNDquery_textuser_nameRedshift自动挂载路径——需要预置RA3集群,并完成四步设置:创建角色(信任策略必须同时包含和,后者需拥有、、、这四个权限)、将角色附加到集群、在Lake Formation中注册表存储桶、将Redshift服务关联角色添加到。最常导致失败的约束条件:
query_s3_tablesredshift.amazonaws.comlakeformation.amazonaws.comsts:AssumeRolests:SetContextsts:SetSourceIdentitysts:TagSessionReadOnlyAdmins- 为两个信任语句添加条件——仅指定服务主体存在混淆代理风险。
aws:SourceAccount - 请勿将附加到集群的查询角色。该权限仅在设置期间由执行设置的主体需要,集群角色仅需读取权限。
AWSLakeFormationDataAdmin - 自动挂载是轮询机制,而非回调——目录可能需要最多300秒才会出现在中。重启集群可强制立即发现目录。
pg_database
4. Identify the Target Table
4. 确定目标表
Namespace — resolve it from the API, do not construct it:
- Read from
S3Tables.S3TableNamespace(Provisioned) ordescribe-logging-statusfroms3TablePublishStatus.s3TableNamespace(Serverless) and use it verbatim.get-namespace - Optional sanity check only: the API value typically follows for
<namespace_arn_id>_sys/clustergranularity andnamespacefor<account>_sysgranularity. Use this only to verify the value looks right — never to generate the namespace when the API response is unavailable.account
Table — each publishable system table maps 1:1 to a table in the table bucket. Do not work from a memorized list — resolve it at runtime, in this order:
aws-redshift- The published set for this cluster/namespace — (Provisioned) or
S3Tables.S3Tables(Serverless) from the status call above, e.g.s3TablePublishStatus.s3Tables. This is the only authoritative answer to "what can I query right now".sys_query_history - The set this API accepts — (accepted
aws redshift enable-logging helpvalues) or--log-exports(acceptedaws redshift-serverless update-namespace helpvalues).--s3-table-names - What each table contains — the public Redshift SYS monitoring views reference, which documents every view and its columns. AWS adds views over time, so treat the docs as the current list rather than hardcoding one.
SYS_*
Column names and types come from the same public reference, or from the live table:
bash
aws glue get-table --region <REGION> \
--catalog-id "<ACCOUNT>:s3tablescatalog/aws-redshift" \
--database-name "<NAMESPACE>" --name "<SYS_TABLE>"Two caveats when reading the public docs against a published table: enum-valued columns (, , ) gain values over time, so confirm with rather than filtering on an assumed set; and the published Iceberg table prepends warehouse-identity columns (, , and peers) that the in-cluster view does not have — they are how you tell apart multiple clusters publishing at granularity.
query_typestatuseventSELECT DISTINCTwarehouse_namewarehouse_namespace_arnSYS_account命名空间——从API响应中获取,请勿自行构造:
- 从(预置集群)的
describe-logging-status或S3Tables.S3TableNamespace(无服务器命名空间)的get-namespace中读取命名空间,并直接使用该值。s3TablePublishStatus.s3TableNamespace - 可选的 sanity check:API返回的值通常遵循以下格式——/
cluster粒度为namespace,<namespace_arn_id>_sys粒度为account。仅用此格式验证返回值是否合理——当API响应不可用时,切勿自行生成命名空间。<account>_sys
表——每个可发布的系统表与表存储桶中的一个表一一对应。请勿依赖记忆的列表——请按以下顺序在运行时确定:
aws-redshift- 此集群/命名空间的已发布表集合——上述状态查询中的(预置集群)或
S3Tables.S3Tables(无服务器命名空间),例如s3TablePublishStatus.s3Tables。这是“我现在可以查询哪些表”的唯一权威答案。sys_query_history - API接受的表集合——运行查看可接受的
aws redshift enable-logging help值,或运行--log-exports查看可接受的aws redshift-serverless update-namespace help值。--s3-table-names - 各表包含的内容——参考公开的Redshift SYS监控视图文档,该文档记录了每个视图及其列。AWS会不断添加新视图,请以文档为准,而非硬编码列表。
SYS_*
列名和类型可从上述公开文档或实时表中获取:
bash
aws glue get-table --region <REGION> \
--catalog-id "<ACCOUNT>:s3tablescatalog/aws-redshift" \
--database-name "<NAMESPACE>" --name "<SYS_TABLE>"对照公开文档查看已发布表时的两个注意事项:枚举值列(、、)会不断新增取值,请使用确认,而非基于假设的集合进行过滤;已发布的Iceberg表会添加集群标识列(、等),而集群内的视图没有这些列——这些列用于区分在粒度下发布数据的多个集群。
query_typestatuseventSELECT DISTINCTwarehouse_namewarehouse_namespace_arnSYS_account5. Query
5. 查询
Query from Athena
从Athena查询
Query syntax:
sql
"s3tablescatalog/aws-redshift"."<NAMESPACE>"."<SYS_TABLE>"查询语法:
sql
"s3tablescatalog/aws-redshift"."<NAMESPACE>"."<SYS_TABLE>"Query from Redshift (Auto-Mounted Catalog)
从Redshift查询(自动挂载目录)
Once the auto-mounted catalog is set up (see ), query using cross-database notation:
${SKILL_DIR}/references/permissions-setup.mdsql
"aws-redshift@s3tablescatalog"."<NAMESPACE>".<SYS_TABLE>完成自动挂载目录的设置后(详见),使用跨数据库语法查询:
${SKILL_DIR}/references/permissions-setup.mdsql
"aws-redshift@s3tablescatalog"."<NAMESPACE>".<SYS_TABLE>Query from Redshift (External Schema)
从Redshift查询(外部模式)
Alternatively, create an external schema pointing to the S3 Tables catalog:
sql
CREATE EXTERNAL SCHEMA <schema_name>
FROM DATA CATALOG
DATABASE '<NAMESPACE>'
CATALOG_ID '<ACCOUNT>:s3tablescatalog/aws-redshift'
IAM_ROLE 'arn:aws:iam::<ACCOUNT>:role/query_s3_tables'
REGION '<REGION>';
SELECT * FROM <schema_name>.<SYS_TABLE> LIMIT 10;或者,创建指向S3 Tables目录的外部模式:
sql
CREATE EXTERNAL SCHEMA <schema_name>
FROM DATA CATALOG
DATABASE '<NAMESPACE>'
CATALOG_ID '<ACCOUNT>:s3tablescatalog/aws-redshift'
IAM_ROLE 'arn:aws:iam::<ACCOUNT>:role/query_s3_tables'
REGION '<REGION>';
SELECT * FROM <schema_name>.<SYS_TABLE> LIMIT 10;Constraints
约束条件
- You MUST run or
describe-logging-statusto get the namespace before writing any SQL query — never construct it manuallyget-namespace - For Athena queries, you MUST confirm workgroup and output location before executing
- Timing columns are in microseconds. Divide by for seconds
1000000.0 - Tables are read-only — no /
INSERT/UPDATEDELETE - Always add a when the user doesn't specify one; filter on
LIMIT/start_timewhere possiblerecord_time
- 编写任何SQL查询前,必须运行或
describe-logging-status获取命名空间——切勿手动构造get-namespace - 执行Athena查询前,必须确认工作组和输出位置已配置
- 时间列以微秒为单位。需除以转换为秒
1000000.0 - 表为只读——不支持/
INSERT/UPDATE操作DELETE - 当用户未指定时,务必添加;尽可能根据
LIMIT/start_time进行过滤record_time
Examples
示例
Worked SQL for the common asks — longest-running queries, error analysis, connection auditing, queue-time trends, cross-table joins — is in . Two rules that apply to every one of them:
${SKILL_DIR}/references/example-queries.md- Timing columns are microseconds. Divide by 1,000,000 for seconds. Reporting as-is overstates durations by 10^6.
elapsed_time - Filter on the Iceberg partition columns (/
year/monthor the table's own partitioning) in addition to any timestamp predicate, or the engine scans the full history.day
常见需求的SQL示例(最长运行时间查询、错误分析、连接审计、队列时间趋势、跨表关联)请参见****。所有示例均需遵循以下两条规则:
${SKILL_DIR}/references/example-queries.md- 时间列以微秒为单位。需除以1,000,000转换为秒。直接使用会将持续时间夸大10^6倍。
elapsed_time - 除了时间戳谓词外,还需根据Iceberg分区列过滤(/
year/month或表自身的分区),否则引擎会扫描全部历史数据。day
Routing: Athena vs Redshift vs Direct SYS_ Access
路由选择:Athena vs Redshift vs 直接访问SYS_
| Scenario | Use |
|---|---|
| Historical/high-volume log analysis, no cluster load | Athena or Redshift on S3 Tables |
| Already connected to a Redshift cluster, want to query S3 Tables logs | Redshift cross-database or external schema |
| Join system table logs with other lake data | Athena or Redshift Spectrum |
| Real-time current state of the cluster | Direct |
| Quick ad-hoc query without Redshift cluster access | Athena |
| 场景 | 使用方式 |
|---|---|
| 历史/高容量日志分析,不希望给集群带来负载 | Athena或Redshift查询S3 Tables |
| 已连接到Redshift集群,希望查询S3 Tables日志 | Redshift跨数据库或外部模式 |
| 将系统表日志与其他湖仓数据关联 | Athena或Redshift Spectrum |
| 集群的实时当前状态 | 直接在集群上查询 |
| 无Redshift集群访问权限的临时即席查询 | Athena |
Key Behaviors
核心特性
- No backfill — only events recorded after enabling are delivered to S3 Tables
- Namespace from the API — always read the namespace from (
describe-logging-status) orS3Tables.S3TableNamespace(get-namespace); never construct it manuallys3TablePublishStatus.s3TableNamespace - Microsecond timing — all duration columns are in microseconds; divide by 1000000.0 for seconds
- Read-only — published tables cannot be written to
- Both Provisioned and Serverless — same table bucket (), different enable APIs
aws-redshift - Any Iceberg-compatible engine — query from Athena, Redshift, or any engine that reads Iceberg
- 无回填——仅会将启用后记录的事件交付到S3 Tables
- 从API获取命名空间——始终从的
describe-logging-status或S3Tables.S3TableNamespace的get-namespace中读取命名空间;切勿手动构造s3TablePublishStatus.s3TableNamespace - 微秒级计时——所有持续时间列均以微秒为单位;需除以1000000.0转换为秒
- 只读——已发布的表无法写入
- 同时支持预置和无服务器环境——使用同一个表存储桶(),但启用API不同
aws-redshift - 兼容任何Iceberg引擎——可从Athena、Redshift或任何支持Iceberg的引擎查询
Troubleshooting
故障排除
| Error | Cause | Fix |
|---|---|---|
| S3 Tables integration not enabled or logging not started | Run |
| S3 Tables not registered in Glue | Enable integration: S3 console > Table buckets > Enable integration |
| Athena table empty after enabling | Ingestion still in flight | Check |
| System table not included when enabling | Re-run enable with that table included, or use |
| Wrong / empty namespace | Namespace constructed instead of read from API | Use the namespace from the describe/get response — |
Status response has no | Outdated AWS CLI / SDK. The field is silently omitted rather than raising an error, so this looks identical to the feature being disabled | Upgrade the CLI/SDK, then re-run. Confirm publishing is actually off before acting on the absence — check the |
| Provisioned flag names used against | Use |
| Missing | See |
Empty results from | Querying identity lacks visibility | Use an identity with superuser-level access |
Catalog doesn't appear in | LF resource not registered, or SLRs not ReadOnlyAdmins | Complete the Lake Formation steps in |
| "Unable to assume role" from Glue | Missing | Fix trust policy and attach |
| Normal after Lake Formation registration | No action needed if queries work |
| Query fails with "does not exist" in Redshift | Catalog not yet auto-mounted (poll delay) | Wait up to 300s or reboot cluster |
| 错误 | 原因 | 修复方案 |
|---|---|---|
| S3 Tables集成未启用或日志发布未启动 | 运行 |
Athena中返回 | S3 Tables未在Glue中注册 | 启用集成:S3控制台 > 表存储桶 > 启用集成 |
| 启用后Athena表为空 | 数据仍在传输中 | 检查 |
命名空间中缺少 | 启用时未包含该系统表 | 重新运行启用命令并包含该表,或使用 |
| 命名空间错误/为空 | 手动构造了命名空间而非从API获取 | 使用describe/get响应中的命名空间—— |
状态响应中完全没有 | AWS CLI/SDK版本过旧。该字段会被静默省略而非报错,因此看起来与功能未启用的状态一致 | 升级CLI/SDK后重新运行。在根据字段缺失采取行动前,请确认日志发布是否真的未开启——检查 |
无服务器环境中返回 | 将预置集群的参数名称用于 | 使用 |
查询表时返回 | 缺少 | 参见 |
| 查询身份无可见性权限 | 使用具有超级用户级别的身份查询 |
目录未出现在 | LF资源未注册,或SLR未加入ReadOnlyAdmins | 完成 |
| Glue返回“Unable to assume role” | 信任策略中缺少 | 修复信任策略并附加 |
返回 | Lake Formation注册后的正常状态 | 若查询可正常执行,则无需操作 |
| Redshift中查询返回“does not exist” | 目录尚未自动挂载(轮询延迟) | 等待最多300秒或重启集群 |
Security Considerations
安全注意事项
Full policies, key policy, and detection setup: . Read it before granting access. The non-negotiables:
${SKILL_DIR}/references/security.md- Scope IAM to the S3 Tables catalog, not wildcards. Glue database/table ARNs nest under — the bare
s3tablescatalog/aws-redshiftform grants metadata read on the whole account.database/*is the one action that must uselakeformation:GetDataAccess; constrain it with an"Resource": "*"aws:ResourceAccountcondition.StringEquals - The KMS key policy needs two principals, not one: (publisher) and
systemtables.redshift.amazonaws.com(compaction). Granting only the publisher lets writes succeed while compaction silently fails.maintenance.s3tables.amazonaws.com - can contain credentials, not just schema — interpolated SQL and
query_textland verbatim inCREATE USER ... PASSWORD. Treat broad access to that table as a secrets-exposure decision; restrict the column with Lake Formation.sys_query_history - Publishing is itself auditable and worth alarming on.
s3tables.amazonaws.comspikes andAccessDeniedfailed-auth counts are the two signals to alert on; encrypt the alarm topic with a customer-managed key.sys_connection_log
完整的策略、密钥策略和检测设置请参见****。授予访问权限前,请先阅读该文件。以下是必须遵守的规则:
${SKILL_DIR}/references/security.md- 将IAM权限限定在S3 Tables目录范围内,而非使用通配符。Glue数据库/表ARN嵌套在下——使用
s3tablescatalog/aws-redshift会授予整个账户的元数据读取权限。database/*是唯一必须使用lakeformation:GetDataAccess的操作;请通过"Resource": "*"的aws:ResourceAccount条件进行约束。StringEquals - KMS密钥策略需要两个主体,而非一个:(发布者)和
systemtables.redshift.amazonaws.com(压缩)。若仅授予发布者权限,写入操作可成功,但压缩操作会静默失败。maintenance.s3tables.amazonaws.com - 可能包含凭证,而非仅包含架构信息——插值SQL和
query_text会原样存储在CREATE USER ... PASSWORD中。请将对该表的广泛访问视为可能泄露密钥的操作;通过Lake Formation限制该列的访问权限。sys_query_history - 日志发布本身可被审计且值得设置告警。的
s3tables.amazonaws.com峰值和AccessDenied中的失败认证次数是两个需要告警的信号;请使用客户托管密钥加密告警主题。sys_connection_log
Reference Files
参考文件
${SKILL_DIR}| File | What it covers | When to load |
|---|---|---|
| Athena prerequisites and workgroup encryption; the full Redshift auto-mount path — IAM role trust/inline policies, Lake Formation | Before running any IAM or Lake Formation setup |
| Worked SQL for longest-running queries, error analysis, connection auditing, queue-time trends, and joins across | When writing queries against the published tables |
| Full least-privilege policy, KMS key policy with both service principals, | Before granting access, or when hardening an existing setup |
${SKILL_DIR}| 文件 | 涵盖内容 | 加载时机 |
|---|---|---|
| Athena先决条件和工作组加密;完整的Redshift自动挂载路径——IAM角色信任/内联策略、Lake Formation的 | 执行任何IAM或Lake Formation设置前 |
| 最长运行时间查询、错误分析、连接审计、队列时间趋势以及跨 | 编写针对已发布表的查询时 |
| 完整的最小权限策略、包含两个服务主体的KMS密钥策略、 | 授予访问权限前,或加固现有设置时 |
Additional Resources
额外资源
Security best practices:
- Amazon Redshift security best practices
- S3 Tables security and access management for S3 Tables
- IAM security best practices
- Lake Formation underlying data access control — why requires
lakeformation:GetDataAccess"Resource": "*"
安全最佳实践:
- Amazon Redshift安全最佳实践
- S3 Tables安全和S3 Tables访问管理
- IAM安全最佳实践
- Lake Formation底层数据访问控制——解释为何需要
lakeformation:GetDataAccess"Resource": "*"