What's running on Model Serving, how much traffic and token throughput each endpoint handles, which endpoints are dormant (paying but idle), and whether AI Gateway is seeing abuse.
The request-level telemetry table for Model Serving endpoints — the fact table of this domain.
request_count column — request volume is COUNT(*).request_time (TIMESTAMP) — when the request was served; drives every date window.workspace_id — workspace the endpoint lives in.served_entity_id — the served entity that handled the request. This table carries served_entity_id, not endpoint_id — endpoint identity is resolved by joining to served_entities.status_code (INTEGER) — HTTP status; 2xx = success, everything else = error.input_token_count / output_token_count (LONG) — per-request token throughput (separate magnitude from request counts, never summed together).SELECT on system.serving.*. An account that has never served a model yields a valid but empty result (queries degrade to not_assessed, never a fake "all endpoints dormant"). Note: two of the older queries here assume alternate column names (request_count, served_entity_input_tokens, served_entity_output_tokens) that are not the confirmed schema — see Notes.The configuration/dimension table describing what each endpoint serves (a model, a foundation-model route, an external model, etc.).
(workspace_id, endpoint_id, served_entity_id) via ROW_NUMBER() … ORDER BY change_time DESC before joining, so a renamed/reconfigured entity is not double-counted.workspace_id, endpoint_id, endpoint_name — endpoint identity (the human-readable name lives here, not in endpoint_usage).served_entity_id, served_entity_name — the served entity within the endpoint (an endpoint can host multiple).entity_type, entity_name, entity_version — what is being served (e.g. a registered model + version, or an external/foundation model).change_time (TIMESTAMP) — config-change timestamp; used both to dedupe and, in the dormant query, as a recency floor (latest_change_time) so a newly-created endpoint inside the window is not mislabeled dormant.endpoint_usage — Model Serving in use, Unity Catalog, SELECT on system.serving.*. entity_creator/creation-flag columns are assumed but not verified.Request telemetry captured by Mosaic AI Gateway — the governance/proxy layer that fronts serving endpoints for rate-limiting, usage tracking, and safety.
request_time (TIMESTAMP), workspace_id, endpoint_name.requester — the calling principal (an individual user or service principal, never a team — do not relabel).status_code (HTTP) — 2xx = success, 429 = rate-limit/quota hit, other = error.request_count, input_tokens, output_tokens — volume and token throughput.request_duration_ms — end-to-end latency (column name uncertain); summarized via percentile_approx(col, 0.5/0.95), deliberately avoiding the PERCENTILE(col, 95) mistake.SELECT on system.ai_gateway.*. Column names above are unverified for this workspace; an absent column degrades that one metric to "unknown" rather than erroring the whole finding. A 429/error row is an abuse/quota signal, not proof of misuse.system.* schemas aren’t uniformly available — several (e.g. system.query, system.lakeflow) are still in preview or must be enabled per-metastore. A query that returns nothing usually means its schema isn’t enabled for your account yet.system.billing.list_prices) unless a query explicitly joins account_prices.A daily per-workspace, per-endpoint, per-requester breakdown of AI Gateway request volume, its success / rate-limited / error split, input/output token throughput, and p50/p95/max latency.
system.ai_gateway.usageOne row = a day + workspace + endpoint + requester's AI Gateway traffic. The columns that matter are total_requests, rate_limited_requests, and error_requests - a rise in rate_limited_requests means you are hitting Gateway or provider rate limits, and a rise in error_requests means calls are failing outright; both mean tokens and latency spent on calls that did not succeed.
RequiresSELECT on system.ai_gateway; must be enabled per-metastore (empty until AI Gateway is enabled on an endpoint)
Returns no rows ifSchema not enabledPreview not rolled outAccount-admin read onlyUsage tracking off
AI Gateway sits in front of serving endpoints precisely to catch abuse, and this query turns its raw telemetry into the signals an admin acts on: a single requester driving runaway token volume (tokens are billable throughput on pay-per-token endpoints), a spike in 429s meaning your rate limits or quotas are being hit, or latency blowups degrading every consumer. Left unwatched, one misconfigured client or leaked service-principal credential can quietly multiply token spend or exhaust quota for legitimate users. Because it slices by requester, it lets you attribute that pressure to a specific user or service principal before it becomes a bill or an outage.
SELECT
CAST(g.event_time AS DATE) AS usage_date,
g.workspace_id AS workspace_id,
CASE WHEN g.endpoint_name IS NULL THEN g.endpoint_name ELSE concat(substr(g.endpoint_name, 1, 2), '****') END AS endpoint_name,
CASE
WHEN g.requester IS NULL OR g.requester = '__REDACTED__' THEN g.requester
WHEN g.requester LIKE '%@%' THEN concat(substr(g.requester, 1, 2), '****@****')
WHEN g.requester RLIKE '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' THEN g.requester
ELSE concat(substr(g.requester, 1, 2), '****')
END AS requester,
COUNT(*) AS total_requests,
-- 2xx == success; 429 == rate-limit/quota; other == error (unconfirmed mapping, see caveats).
SUM(CASE WHEN g.status_code BETWEEN 200 AND 299 THEN 1 ELSE 0 END) AS success_requests,
SUM(CASE WHEN g.status_code = 429 THEN 1 ELSE 0 END) AS rate_limited_requests,
SUM(CASE WHEN g.status_code IS NOT NULL
AND NOT (g.status_code BETWEEN 200 AND 299)
AND g.status_code <> 429 THEN 1 ELSE 0 END) AS error_requests,
-- Token throughput is its own magnitude, never blended with request counts.
SUM(COALESCE(g.input_tokens, 0)) AS input_tokens,
SUM(COALESCE(g.output_tokens, 0)) AS output_tokens,
-- Latency via percentile_approx (fraction form).
percentile_approx(g.latency_ms, 0.5) AS p50_latency_ms,
percentile_approx(g.latency_ms, 0.95) AS p95_latency_ms,
MAX(g.latency_ms) AS max_latency_ms
FROM system.ai_gateway.usage g
WHERE g.event_time >= dateadd(day, -:period_days, current_date())
AND g.event_time < current_date()
GROUP BY
CAST(g.event_time AS DATE),
g.workspace_id,
g.endpoint_name,
g.requester
ORDER BY usage_date DESC, total_requests DESC:period_daysdefault 30rolling window in days| Column | How to read it |
|---|---|
| usage_date | The calendar date (from event_time) the requests were served. One row per day within the window; today is excluded. |
| workspace_id | The workspace the gateway-fronted endpoint lives in. Use as a join key only within its own workspace scope. |
| endpoint_name | The serving endpoint name, masked to its first 2 characters + '****'. NULL is passed through untouched (not masked). |
| requester | The individual calling principal — a user or a service principal, never a team. Masked by type: emails to 'xx****@****', raw GUID service principals left intact, other principals truncated, and NULL / '__REDACTED__' sentinels passed through. Do not relabel this as a team. |
| total_requests | Total requests in the slice (COUNT(*); system.ai_gateway.usage is one row per request keyed by request_id — there is no request_count column). Integer count, not DBUs or dollars. |
| success_requests | Requests with a 2xx status_code (counted as rows). The 2xx-as-success mapping holds; only the 429 rate-limit literal remains assumed. |
| rate_limited_requests | Requests with status_code 429 — the assumed rate-limit/quota-hit literal. A spike here is a signal to investigate quotas, not proof of misuse. |
| error_requests | Requests with a non-null status code that is neither 2xx nor 429 — everything else that failed. |
| input_tokens | Total input (prompt) tokens across the slice (nulls treated as 0). A throughput magnitude, kept separate from request counts — never blend the two. |
| output_tokens | Total output (completion) tokens across the slice (nulls treated as 0). Separate magnitude from request counts. |
| p50_latency_ms | Approximate median end-to-end latency in milliseconds (percentile_approx, fraction 0.5) over the latency_ms column. |
| p95_latency_ms | Approximate 95th-percentile latency in milliseconds (percentile_approx, fraction 0.95) over the latency_ms column — the tail experienced by the slowest 5% of requests. |
| max_latency_ms | The single slowest request's end-to-end latency in the slice, in milliseconds. |
Inventory / reference query — it returns state to read, not a pass/fail band.
| usage_date | endpoint_name | requester | total_requests | rate_limited_requests | error_requests | output_tokens | p95_latency_ms |
|---|---|---|---|---|---|---|---|
| 2026-07-05 | pr**** | da****@**** | 48120 | 3200 | 410 | 9840000 | 1450 |
| 2026-07-05 | ll**** | 4f8a1c2d-1b3e-4a5f-9c7d-2e6f0a1b3c4d | 1205 | 0 | 2 | 240500 | 880 |
| 2026-07-04 | pr**** | sv**** | 96 | 0 | 0 | 12800 | 310 |
Sort by rate_limited_requests, error_requests, or output_tokens to find the requester x endpoint slices under abuse or quota pressure; identify the specific user or service principal driving it, then adjust the endpoint's rate limits/quotas or revoke the offending principal, and cross-reference system.billing.usage to quantify the DBU/dollar impact. Before acting on any number, confirm the ai_gateway.usage column names against this workspace.
Feeds into: compute_ai_gateway_abuse (gov-6) — AI Gateway request volume, error-rate spikes, latency, token throughput, rate-limit/quota hits
One row per endpoint that <em>billed</em> in the window — its DBUs, a list-price dollar estimate, request volume, and a <code>tracking_status</code> that tells a truly idle endpoint apart from one whose usage tracking is simply switched off.
system.billing.usagesystem.serving.endpoint_usagesystem.serving.served_entitiesOne row = one endpoint that BILLED in the window (any serving product, including Vector Search), with its DBUs, a list-price dollar estimate, request volume, and a tracking_status. The columns that matter are est_usd_list (spend) and tracking_status / status - they separate "usage tracking is OFF" (spend but no telemetry) from "truly idle" (tracked but zero requests).
RequiresSELECT on system.billing, system.serving, system.access; the serving and access system schemas enabled; billing is GA, serving + workspaces_latest are Public Preview
Returns no rows ifSchema not enabledPreview not rolled outUsage tracking off
An idle serving endpoint still bills for its reserved capacity, so it is prime waste — but “zero requests” is ambiguous. It can mean the endpoint is genuinely idle, that its AI Gateway usage tracking is switched off (so real traffic is invisible), or that it is a Vector Search endpoint that never writes to the serving tables at all. Kill the wrong one and you take down something that was quietly serving production traffic.
This query anchors on system.billing.usage — so it sees every endpoint that billed, Model Serving and Vector Search alike — then classifies each into a tracking_status and a worst-first status band. It puts exact DBUs and a list-price dollar estimate next to each, so you can rank the waste by money and act on the CRITICAL-idle rows with confidence instead of guessing.
WITH entities AS ( -- served_entities is change-history -> collapse to the latest config row per entity
SELECT workspace_id, endpoint_id, endpoint_name, served_entity_id, served_entity_name,
entity_type, entity_name, entity_version, latest_change_time
FROM (
SELECT se.workspace_id, se.endpoint_id, se.endpoint_name, se.served_entity_id,
se.served_entity_name, se.entity_type, se.entity_name, se.entity_version,
MAX(se.change_time) OVER (PARTITION BY se.workspace_id, se.endpoint_id, se.served_entity_id) AS latest_change_time,
ROW_NUMBER() OVER (PARTITION BY se.workspace_id, se.endpoint_id, se.served_entity_id ORDER BY se.change_time DESC) AS _rn
FROM system.serving.served_entities se
) WHERE _rn = 1
),
ent_map AS ( -- served_entity_id -> endpoint_id, so endpoint_usage (which has no endpoint_id) can roll up to endpoint level
SELECT DISTINCT workspace_id, endpoint_id, served_entity_id FROM entities
),
usage_entity AS ( -- per served entity, inside the analysis window
SELECT eu.workspace_id, eu.served_entity_id,
COUNT(*) AS total_requests,
MAX(CAST(eu.request_time AS DATE)) AS last_request_date
FROM system.serving.endpoint_usage eu
WHERE eu.request_time >= dateadd(day, -:period_days, current_date())
AND eu.request_time < current_date()
GROUP BY eu.workspace_id, eu.served_entity_id
),
usage_endpoint AS ( -- requests rolled up to endpoint, inside the analysis window
SELECT m.workspace_id, m.endpoint_id,
SUM(ue.total_requests) AS ep_requests,
MAX(ue.last_request_date) AS ep_last_request_date
FROM usage_entity ue
JOIN ent_map m ON ue.workspace_id = m.workspace_id AND ue.served_entity_id = m.served_entity_id
GROUP BY m.workspace_id, m.endpoint_id
),
usage_ever AS ( -- ANY tracked row over the full retention window -> distinguishes "tracking off" from "idle in window"
SELECT m.workspace_id, m.endpoint_id, COUNT(*) AS ever_requests
FROM system.serving.endpoint_usage eu
JOIN ent_map m ON eu.workspace_id = m.workspace_id AND eu.served_entity_id = m.served_entity_id
WHERE eu.request_time >= dateadd(day, -:retention_days, current_date())
GROUP BY m.workspace_id, m.endpoint_id
),
price AS (
SELECT sku_name, cloud, usage_unit, price_start_time, price_end_time,
CAST(pricing.default AS DOUBLE) AS list_rate
FROM system.billing.list_prices
),
cost AS ( -- billing per (workspace, endpoint, product); VS carries endpoint_name, model serving carries endpoint_id -> COALESCE
SELECT u.workspace_id,
u.usage_metadata.endpoint_id AS endpoint_id,
u.usage_metadata.endpoint_name AS endpoint_name_billing,
u.billing_origin_product AS product,
SUM(u.usage_quantity) AS net_dbus,
SUM(u.usage_quantity * COALESCE(p.list_rate, 0)) AS est_usd_list
FROM system.billing.usage u
LEFT JOIN price p
ON u.sku_name = p.sku_name AND u.cloud = p.cloud AND u.usage_unit = p.usage_unit
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE upper(u.usage_unit) = 'DBU'
AND (u.usage_metadata.endpoint_id IS NOT NULL OR u.usage_metadata.endpoint_name IS NOT NULL)
AND u.usage_date >= dateadd(day, -:period_days, current_date())
AND u.usage_date < current_date()
GROUP BY u.workspace_id, u.usage_metadata.endpoint_id, u.usage_metadata.endpoint_name, u.billing_origin_product
),
cost_ep AS ( -- collapse products -> one row per endpoint (total cost), keep the product list
SELECT workspace_id,
COALESCE(endpoint_id, endpoint_name_billing) AS endpoint_key,
MAX(endpoint_id) AS endpoint_id,
MAX(endpoint_name_billing) AS endpoint_name_billing,
SUM(net_dbus) AS net_dbus,
SUM(est_usd_list) AS est_usd_list,
array_join(collect_set(product), ', ') AS products,
MAX(CASE WHEN product = 'VECTOR_SEARCH' THEN 1 ELSE 0 END) AS is_vs
FROM cost GROUP BY workspace_id, COALESCE(endpoint_id, endpoint_name_billing)
)
SELECT
ce.workspace_id,
w.workspace_name,
ce.endpoint_id,
CASE WHEN COALESCE(ent.endpoint_name, ce.endpoint_name_billing) IS NULL THEN NULL
ELSE concat(substr(COALESCE(ent.endpoint_name, ce.endpoint_name_billing), 1, 2), '****') END AS endpoint_name,
ce.products,
ent.served_entity_id,
CASE WHEN ent.served_entity_name IS NULL THEN NULL
ELSE concat(substr(ent.served_entity_name, 1, 2), '****') END AS served_entity_name,
ent.entity_type, -- FOUNDATION_MODEL / CUSTOM_MODEL / EXTERNAL_MODEL / FEATURE_SPEC ...
ent.entity_name,
ent.entity_version,
ent.latest_change_time,
ROUND(ce.net_dbus, 1) AS net_dbus,
ROUND(ce.est_usd_list, 2) AS est_usd_list,
COALESCE(ue.total_requests, 0) AS entity_requests_window,
ue.last_request_date AS entity_last_request_date,
COALESCE(uep.ep_requests, 0) AS endpoint_requests_window,
uep.ep_last_request_date,
-- Is the "0 requests" real, or just untracked? (free-text detail; the enum band is `status`)
CASE
WHEN ce.is_vs = 1 THEN 'Vector Search - no serving-table telemetry (see cost_vector_search_spend)'
WHEN ent.endpoint_id IS NULL THEN 'bills but not in served_entities (not a tracked model-serving endpoint)'
WHEN COALESCE(uev.ever_requests, 0) = 0 AND ce.net_dbus > 0 THEN 'usage tracking likely OFF - spend but zero tracked rows in retention'
WHEN COALESCE(uep.ep_requests, 0) = 0 THEN 'tracking ON - idle in window (had traffic within retention)'
ELSE 'tracking ON - active'
END AS tracking_status,
CASE
WHEN ce.is_vs = 1 THEN 'NOT_ASSESSED' -- Vector Search: no serving telemetry
WHEN ent.endpoint_id IS NULL THEN 'NOT_ASSESSED' -- bills but not a tracked model-serving endpoint
WHEN COALESCE(uev.ever_requests, 0) = 0 AND ce.net_dbus > 0 THEN 'WARN' -- spend but tracking off: enable tracking, do not call it idle
WHEN COALESCE(uep.ep_requests, 0) = 0 AND ce.net_dbus > 0 THEN 'CRITICAL' -- tracked + spend + zero requests = truly idle
WHEN COALESCE(uep.ep_requests, 0) <= :warn_low_requests THEN 'WARN'
ELSE 'OK'
END AS status
FROM cost_ep ce
LEFT JOIN entities ent ON ent.workspace_id = ce.workspace_id AND ent.endpoint_id = ce.endpoint_id
LEFT JOIN usage_entity ue ON ue.workspace_id = ent.workspace_id AND ue.served_entity_id = ent.served_entity_id
LEFT JOIN usage_endpoint uep ON uep.workspace_id = ce.workspace_id AND uep.endpoint_id = ce.endpoint_id
LEFT JOIN usage_ever uev ON uev.workspace_id = ce.workspace_id AND uev.endpoint_id = ce.endpoint_id
LEFT JOIN system.access.workspaces_latest w ON w.workspace_id = ce.workspace_id
ORDER BY
CASE status WHEN 'CRITICAL' THEN 0 WHEN 'WARN' THEN 1 WHEN 'NOT_ASSESSED' THEN 2 ELSE 3 END,
ce.net_dbus DESC, endpoint_requests_window ASC
LIMIT :top_n:period_daysdefault 30analysis window in days for requests and cost:retention_daysdefault 90the full endpoint_usage retention window, used to tell "usage tracking off" from "idle in window":warn_low_requestsdefault 10endpoint requests over the window at or below which a billed endpoint flags WARN:top_ndefault 200row capsystem.billing.usage so it captures every endpoint that billed in the window — Model Serving and Vector Search — rather than only the ones that emit serving telemetry.endpoint_usage (which has no endpoint_id and no request count) to the endpoint, via the deduped served_entities dimension, over the trailing :period_days window.:retention_days” signal so a billed endpoint with zero tracked rows reads as usage tracking off, not as idle.tracking_status (active · idle-in-window · tracking-off · Vector-Search-no-telemetry · bills-but-untracked) and a worst-first status band: CRITICAL (tracked, still billing, zero requests = real idle waste), WARN (very low traffic, or spend with tracking off), NOT_ASSESSED (Vector Search / untracked), OK.est · at list) so the waste shows in DBUs and directional dollars — never your negotiated invoice rate.| Column | How to read it |
|---|---|
| workspace_id | The workspace the endpoint bills in. Part of the identity key. |
| workspace_name | Human-readable workspace name (LEFT JOIN to system.access.workspaces_latest, Public Preview) — may be NULL if that table is not enabled. |
| endpoint_id | Stable endpoint identifier (unmasked) — use it to locate and act on the endpoint. NULL for products that bill by name only (e.g. some Vector Search rows). |
| endpoint_name | Endpoint name, masked to its first 2 characters + '****'. Orientation only. |
| products | The billing products this endpoint drew on (e.g. MODEL_SERVING, VECTOR_SEARCH), comma-joined — how you spot a Vector Search endpoint. |
| entity_type | What is served — FOUNDATION_MODEL / CUSTOM_MODEL / EXTERNAL_MODEL / FEATURE_SPEC … NULL when the endpoint bills but is absent from served_entities. |
| net_dbus | Exact billed DBUs (usage_unit='DBU') for this endpoint over :period_days, netted across record types. DBUs, NOT dollars. Endpoint-level — do not sum across the products string. |
| est_usd_list | List-price USD ESTIMATE = net_dbus × list_prices.pricing.default (‘est · at list’). NOT your negotiated invoice, DBU-only, excludes cloud infra / egress — directional only. |
| endpoint_requests_window | Requests this endpoint served in the :period_days window (COUNT of endpoint_usage rows rolled up to the endpoint); 0 is the idle signal — but read it together with tracking_status. |
| ep_last_request_date | Date of the endpoint's most recent tracked request in the window; NULL means no tracked traffic in :period_days (not necessarily “never”). |
| tracking_status | Free-text read on WHY requests are zero: active, idle in window (tracked, had traffic within retention), usage tracking likely OFF (spend but zero tracked rows in retention), Vector Search — no serving-table telemetry, or bills but not in served_entities. A heuristic, not a Databricks flag. |
| status | The worst-first band: CRITICAL (tracked + spend + zero requests = idle waste), WARN (very low requests, or spend with tracking off), NOT_ASSESSED (Vector Search / untracked — can't judge), OK. Field heuristic; tune :warn_low_requests. |
status = OK - the endpoint has request volume proportional to its spend.
status = CRITICAL (tracked, still billing, zero requests in the window = idle waste) or WARN (very low requests, or spend with usage tracking off so idle cannot be confirmed) - field heuristic; tune :warn_low_requests for your account.
| endpoint_name | products | entity_type | net_dbus | est_usd_list | endpoint_requests_window | tracking_status | status |
|---|---|---|---|---|---|---|---|
| ch**** | MODEL_SERVING | FOUNDATION_MODEL | 12800 | 6400.00 | 184220 | tracking ON - active | OK |
| fr**** | MODEL_SERVING | CUSTOM_MODEL | 3100 | 1550.00 | 0 | tracking ON - idle in window (had traffic within retention) | CRITICAL |
| re**** | MODEL_SERVING | EXTERNAL_MODEL | 480 | 240.00 | 0 | usage tracking likely OFF - spend but zero tracked rows in retention | WARN |
| vs**** | VECTOR_SEARCH | (null) | 920 | 460.00 | 0 | Vector Search - no serving-table telemetry (see cost_vector_search_spend) | NOT_ASSESSED |
system.billing.usage, then LEFT JOINs the serving tables — which only cover Model Serving endpoints with AI Gateway usage tracking enabled. So tracking_status reports NOT_ASSESSED for Vector Search (no serving telemetry exists) and for endpoints that bill but are absent from served_entities.:retention_days window is flagged WARN “tracking off” — enable tracking before calling it idle. Set :retention_days to roughly your endpoint_usage retention (~90 days) or real idle gets misread.endpoint_usage has NO endpoint_id and NO request-count column — requests are COUNT(*) per served_entity_id, mapped to the endpoint through served_entities. served_entities is change-history, deduped to the latest row per (workspace_id, endpoint_id, served_entity_id) before the join.tracking_status is a HEURISTIC inferred from whether a billed endpoint has any serving telemetry — it is not a Databricks-reported flag (confidence: needs_confirmation). Verify against your own workspace before acting on borderline rows.usage_quantity × list_prices.pricing.default — a LIST-PRICE ESTIMATE (‘est · at list’), DBU-only, excludes cloud infra / egress $, and is NOT your negotiated invoice rate (no system table carries that).net_dbus and est_usd_list are ENDPOINT-level totals — do NOT sum them across the products list (double-count).endpoint_name and served_entity_name are partial-masked (first 2 chars + '****'); IDs and entity_name pass through intact. Output is still sensitive (workspace IDs, endpoint IDs, spend).system.serving.* and system.access.workspaces_latest are Public Preview and opt-in — the query is empty until the serving + access schemas are enabled and Model Serving is in use; workspace_name may be NULL.usage_date / request_time >= current_date - :period_days AND < current_date — whole days, excludes today's partial data.cost_vector_search_spend, which reads billing_origin_product = 'VECTOR_SEARCH' directly.Work the list worst-first. Scale-to-zero or delete anything flagged CRITICAL idle; for WARN “tracking off” rows, turn on AI Gateway usage tracking so idle can actually be measured before you touch the endpoint; and for the Vector Search rows (NOT_ASSESSED here) cross to cost_vector_search_spend for their real cost and traffic. Right-size provisioned throughput, or move a low-traffic endpoint to pay-per-token.
Feeds into: per-ENDPOINT serving spend (DBUs + list-$) paired with request volume and a tracking_status band — the idle / usage-tracking-off finding that flags CRITICAL wasted endpoints and hands Vector Search rows off to cost_vector_search_spend
A daily rollup of Model Serving traffic per endpoint and served entity, showing total, successful and errored requests plus input/output token totals, with endpoint and entity names joined from the serving configuration.
system.serving.endpoint_usageOne row = a day + endpoint + served entity's request volume. The columns that matter are total_requests, error_requests, and net_dbus - use this as the detail behind a dormancy or cost finding, not as a standalone health signal (this file mixes several metrics with no single waste driver).
RequiresSELECT on system.serving, system.billing; system.serving must be enabled per-metastore (empty until Model Serving is in use); system.billing is GA.
Returns no rows ifSchema not enabledPreview not rolled outUsage tracking off
This is the workhorse baseline of serving activity: it lets an admin spot endpoints with abnormal error rates (a reliability and possibly abuse signal) and surprising token throughput (the real cost driver for pay-per-token serving) before those turn into a bill surprise. Because a single high-traffic or runaway endpoint can dominate serving spend, seeing per-day request and token volume per entity tells you where to focus right-sizing and governance. Note these are request and token counters only — not DBUs or dollars — so they tell you where the money is likely going, not the figure itself.
WITH entities AS (
-- served_entities is change-history; keep only the latest config row per entity.
SELECT
workspace_id,
endpoint_id,
endpoint_name,
served_entity_id,
served_entity_name,
entity_type,
entity_name,
entity_version
FROM (
SELECT
se.*,
ROW_NUMBER() OVER (
PARTITION BY se.workspace_id, se.endpoint_id, se.served_entity_id
ORDER BY se.change_time DESC
) AS _rn
FROM system.serving.served_entities se
)
WHERE _rn = 1
),
price AS (
SELECT sku_name, cloud, usage_unit, price_start_time, price_end_time,
CAST(pricing.default AS DOUBLE) AS list_rate
FROM system.billing.list_prices
),
cost_rollup AS (
-- Pre-aggregated billing cost per serving endpoint per day (billing key = usage_metadata.endpoint_id).
-- Grouped by (workspace_id, endpoint_id, usage_date) to match this query's endpoint/day join grain;
-- billing.usage cannot break cost down to served_entity_id. Unique per key so the LEFT JOIN below is safe.
SELECT u.workspace_id,
u.usage_metadata.endpoint_id AS endpoint_id,
u.usage_date AS usage_date,
SUM(u.usage_quantity) AS net_dbus,
SUM(u.usage_quantity * COALESCE(p.list_rate, 0)) AS est_usd_list
FROM system.billing.usage u
LEFT JOIN price p
ON u.sku_name = p.sku_name AND u.cloud = p.cloud AND u.usage_unit = p.usage_unit
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE upper(u.usage_unit) = 'DBU'
AND u.usage_metadata.endpoint_id IS NOT NULL
AND u.usage_date >= dateadd(day, -:period_days, current_date())
AND u.usage_date < current_date()
GROUP BY u.workspace_id, u.usage_metadata.endpoint_id, u.usage_date
),
finding AS (
SELECT
CAST(eu.request_time AS DATE) AS usage_date,
eu.workspace_id AS workspace_id,
ent.endpoint_id AS endpoint_id,
CASE WHEN ent.endpoint_name IS NULL THEN ent.endpoint_name ELSE concat(substr(ent.endpoint_name, 1, 2), '****') END AS endpoint_name,
eu.served_entity_id AS served_entity_id,
CASE WHEN ent.served_entity_name IS NULL THEN ent.served_entity_name ELSE concat(substr(ent.served_entity_name, 1, 2), '****') END AS served_entity_name,
ent.entity_type AS entity_type,
ent.entity_name AS entity_name,
-- status_code is the HTTP-status column on endpoint_usage; classify 2xx as success,
-- everything else as error. endpoint_usage is one row per request, so counts are
-- COUNT(*)/CASE-based rather than a summed request_count column.
SUM(CASE WHEN eu.status_code BETWEEN 200 AND 299 THEN 1 ELSE 0 END) AS success_requests,
SUM(CASE WHEN eu.status_code IS NOT NULL AND NOT (eu.status_code BETWEEN 200 AND 299)
THEN 1 ELSE 0 END) AS error_requests,
COUNT(*) AS total_requests,
-- Token throughput is a separate magnitude from request counts (never summed with them).
SUM(COALESCE(eu.input_token_count, 0)) AS input_tokens,
SUM(COALESCE(eu.output_token_count, 0)) AS output_tokens
FROM system.serving.endpoint_usage eu
LEFT JOIN entities ent
ON eu.workspace_id = ent.workspace_id
AND eu.served_entity_id = ent.served_entity_id
WHERE eu.request_time >= dateadd(day, -:period_days, current_date())
AND eu.request_time < current_date()
GROUP BY
CAST(eu.request_time AS DATE),
eu.workspace_id,
ent.endpoint_id,
ent.endpoint_name,
eu.served_entity_id,
ent.served_entity_name,
ent.entity_type,
ent.entity_name
)
SELECT
finding.usage_date,
finding.workspace_id,
finding.endpoint_id,
finding.endpoint_name,
finding.served_entity_id,
finding.served_entity_name,
finding.entity_type,
finding.entity_name,
finding.success_requests,
finding.error_requests,
finding.total_requests,
finding.input_tokens,
finding.output_tokens,
-- cost columns (endpoint/day grain; repeats across served-entity rows of the same endpoint/day).
COALESCE(cr.net_dbus, 0) AS net_dbus,
COALESCE(cr.est_usd_list, 0) AS est_usd_list
FROM finding
LEFT JOIN cost_rollup cr
ON finding.workspace_id = cr.workspace_id
AND finding.endpoint_id = cr.endpoint_id
AND finding.usage_date = cr.usage_date
ORDER BY usage_date DESC, total_requests DESC:period_daysdefault 30rolling window in days:period_days window (excluding today).entity_type/entity_name, from the latest serving configuration so the traffic is labeled with what it's actually serving.net_dbus) and a list-price USD estimate (est_usd_list) per endpoint, so the finding now shows the waste in DBUs and directional dollars — the estimate prices DBUs at list, not your negotiated rate.| Column | How to read it |
|---|---|
| usage_date | The calendar day (from request_time) the traffic was served; one row per day within the trailing window, excluding today. |
| workspace_id | The workspace the endpoint lives in; part of the identity used to resolve endpoint and entity names. |
| endpoint_id | The serving endpoint identifier for this row; endpoint_usage carries only served_entity_id, so endpoint_id is resolved from the served_entities dimension and is also the billing-cost join key. |
| endpoint_name | Human-readable endpoint name, MASKED to first 2 chars + '****' (joined from served_entities); NULL if no matching config row. |
| served_entity_id | The served entity within the endpoint that actually handled the request (an endpoint can host several). |
| served_entity_name | Human-readable served-entity name, MASKED to first 2 chars + '****'; NULL if unmatched. |
| entity_type | What is being served (e.g. a registered model, an external model, a foundation-model route), from the latest config row. |
| entity_name | The name of the underlying entity being served (e.g. the model name), from the latest config row; NOT masked. |
| success_requests | Count of requests with a 2xx status_code on this day (COUNT-based; endpoint_usage is one row per request). A request count, not dollars. |
| error_requests | Count of requests with a non-2xx (but non-null) status_code; the error signal for this endpoint-day. |
| total_requests | All requests served on this day (COUNT(*); success + error + any un-classified) — the raw traffic volume. |
| input_tokens | Total input/prompt tokens served on this day (separate magnitude from request counts; a token count, not dollars). |
| output_tokens | Total output/completion tokens served on this day (separate magnitude; a token count, not dollars). |
| net_dbus | Exact billed DBUs (usage_unit='DBU') attributed to this endpoint per day (workspace_id + endpoint_id + usage_date) over the :period_days window, netted across all record_types (COALESCE 0 when no billing row matched). DBUs, NOT dollars. ENDPOINT/day-level: billing cannot see served_entity_id, so the same value REPEATS across served-entity rows of the same endpoint/day — do NOT SUM across those rows (dedupe on endpoint_id+usage_date first). |
| est_usd_list | List-price USD ESTIMATE = net_dbus × list_prices.pricing.default (‘est · at list’). NOT your negotiated invoice rate (not in any system table) and excludes cloud infra/egress $ — directional only, never a real dollar figure. |
Inventory / reference query — it returns state to read, not a pass/fail band.
| usage_date | endpoint_name | entity_name | success_requests | error_requests | total_requests | output_tokens | net_dbus | est_usd_list |
|---|---|---|---|---|---|---|---|---|
| 2026-07-03 | pr**** | llama-3-70b-instruct | 48210 | 915 | 49125 | 3820441 | 8600 | 4300.00 |
| 2026-07-03 | in**** | fraud-scorer | 12004 | 6 | 12010 | 0 | 3200 | 1600.00 |
| 2026-07-04 | pr**** | llama-3-70b-instruct | 51002 | 4488 | 55490 | 4110233 | 9100 | 4550.00 |
Sort the output by error_requests / total_requests to surface high-error-rate endpoints, and by input_tokens + output_tokens to spot surprising token throughput; then confirm the endpoint_usage column names and the endpoint_id join against your workspace and cross to system.billing.usage for the DBU/dollar figure before acting on any endpoint.
Feeds into: compute_serving_endpoint_health (gov-5) — per-endpoint request volume, success vs error, token throughput
One row per Model Serving endpoint over the trailing window, giving total requests, the success/error split, input and output token totals, and the date each endpoint was last hit.
system.serving.endpoint_usagesystem.serving.served_entitiesOne row = one endpoint's total traffic over the window. The columns that matter are request_count and last_request_date - use this as the join input for a cost-per-request or dormancy check, not as a standalone health signal.
RequiresSELECT on system.serving, system.billing; system.serving must be enabled per-metastore (empty until Model Serving is in use); system.billing is GA.
Returns no rows ifSchema not enabledPreview not rolled outUsage tracking off
Model Serving is billed by mode — provisioned-throughput endpoints cost the same whether they handle a million requests or ten, while pay-per-token endpoints scale with usage. Putting real request volume next to each endpoint lets an admin catch cost-mode mismatches: a high-traffic endpoint stuck on pay-per-token, or a low-traffic one parked on expensive provisioned throughput. The error split and last-request date also flag endpoints that are failing or effectively idle before the bill confirms the waste.
WITH entities AS (
SELECT workspace_id, endpoint_id, endpoint_name, served_entity_id
FROM (
SELECT
se.workspace_id, se.endpoint_id, se.endpoint_name, se.served_entity_id,
ROW_NUMBER() OVER (
PARTITION BY se.workspace_id, se.endpoint_id, se.served_entity_id
ORDER BY se.change_time DESC
) AS _rn
FROM system.serving.served_entities se
)
WHERE _rn = 1
),
price AS (
SELECT sku_name, cloud, usage_unit, price_start_time, price_end_time,
CAST(pricing.default AS DOUBLE) AS list_rate
FROM system.billing.list_prices
),
cost_rollup AS (
-- Pre-aggregated to EXACTLY this query's join grain (endpoint_id) so the LEFT JOIN is strictly 1:1
-- and never multiplies result rows. endpoint_id is a globally-unique GUID -> keyed on id alone.
SELECT
u.usage_metadata.endpoint_id AS endpoint_id,
SUM(u.usage_quantity) AS net_dbus,
SUM(u.usage_quantity * COALESCE(p.list_rate, 0)) AS est_usd_list
FROM system.billing.usage u
LEFT JOIN price p
ON u.sku_name = p.sku_name AND u.cloud = p.cloud AND u.usage_unit = p.usage_unit
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE upper(u.usage_unit) = 'DBU'
AND u.usage_metadata.endpoint_id IS NOT NULL
AND u.usage_date >= dateadd(day, -:period_days, current_date())
AND u.usage_date < current_date()
GROUP BY u.usage_metadata.endpoint_id
)
SELECT
ent.endpoint_id AS endpoint_id,
CASE WHEN ent.endpoint_name IS NULL THEN ent.endpoint_name ELSE concat(substr(ent.endpoint_name, 1, 2), '****') END AS endpoint_name,
COUNT(*) AS request_count,
SUM(CASE WHEN eu.status_code BETWEEN 200 AND 299 THEN 1 ELSE 0 END) AS success_requests,
SUM(CASE WHEN eu.status_code IS NOT NULL AND NOT (eu.status_code BETWEEN 200 AND 299) THEN 1 ELSE 0 END) AS error_requests,
SUM(COALESCE(eu.input_token_count, 0)) AS input_tokens,
SUM(COALESCE(eu.output_token_count, 0)) AS output_tokens,
MAX(CAST(eu.request_time AS DATE)) AS last_request_date,
-- cost columns: rollup is constant within each endpoint group (1:1), surfaced via MAX() so the
-- existing per-endpoint grain and COUNT(*) semantics are unchanged.
MAX(COALESCE(cr.net_dbus, 0)) AS net_dbus,
MAX(COALESCE(cr.est_usd_list, 0)) AS est_usd_list
FROM system.serving.endpoint_usage eu
LEFT JOIN entities ent
ON eu.workspace_id = ent.workspace_id
AND eu.served_entity_id = ent.served_entity_id
LEFT JOIN cost_rollup cr
ON ent.endpoint_id = cr.endpoint_id
WHERE eu.request_time >= dateadd(day, -:period_days, current_date())
AND eu.request_time < current_date()
GROUP BY ent.endpoint_id, ent.endpoint_name
ORDER BY request_count DESC, endpoint_id:period_daysdefault 30rolling window in daysinput_tokens and output_tokens separately per endpoint — the throughput signal that drives pay-per-token spend (kept distinct from request counts, never blended).net_dbus) and a list-price USD estimate (est_usd_list) per endpoint, so the finding now shows the waste in DBUs and directional dollars — the estimate prices DBUs at list, not your negotiated rate.| Column | How to read it |
|---|---|
| endpoint_id | Stable identifier of the serving endpoint, resolved from the served_entities dimension. Use it as the join key back to billing or config; it survives renames. Can be NULL for usage rows that matched no config row (unresolved traffic). |
| endpoint_name | Human-readable endpoint name, masked to its first 2 characters + '****' (NULL passes through as NULL). Orienting label only, not for exact matching. |
| request_count | Total inference requests to this endpoint in the window (COUNT of request rows). This is request volume, not DBUs or dollars. |
| success_requests | Requests that returned an HTTP 2xx status. success_requests + error_requests can be less than request_count when status_code is NULL. |
| error_requests | Requests with a non-null, non-2xx status — the failure signal. A high ratio to request_count means the endpoint is erroring. |
| input_tokens | Sum of per-request input token counts (nulls treated as 0). Prompt-side throughput; a separate magnitude from request_count, never added to output_tokens. |
| output_tokens | Sum of per-request output token counts (nulls treated as 0). Generation-side throughput; the main driver of pay-per-token cost. |
| last_request_date | Calendar date of the most recent request in the window. A stale date (well before yesterday) flags an endpoint that has gone dormant. |
| net_dbus | Exact billed DBUs (usage_unit='DBU') attributed to this endpoint (billing endpoint_id) over the :period_days window, netted across all record_types (COALESCE 0 when no billing row matched). DBUs, NOT dollars. Endpoint-level: billing cannot split serving DBUs per served entity, so this matches the per-endpoint grain. |
| est_usd_list | List-price USD ESTIMATE = net_dbus × list_prices.pricing.default (‘est · at list’). NOT your negotiated invoice rate (not in any system table) and excludes cloud infra/egress $ — directional only, never a real dollar figure. |
Inventory / reference query — it returns state to read, not a pass/fail band.
| endpoint_id | endpoint_name | request_count | success_requests | error_requests | input_tokens | output_tokens | last_request_date | net_dbus | est_usd_list |
|---|---|---|---|---|---|---|---|---|---|
| a1b2c3d4-0000-4a1b-8c2d-111122223333 | pr**** | 48210 | 47995 | 215 | 6821440 | 1930875 | 2026-07-05 | 9200 | 4600.00 |
| f9e8d7c6-0000-4f9e-8d7c-444455556666 | ll**** | 912 | 900 | 12 | 1284500 | 402118 | 2026-07-05 | 1400 | 700.00 |
| 0c1d2e3f-0000-40c1-8d2e-777788889999 | de**** | 37 | 30 | 7 | 51200 | 8940 | 2026-06-14 | 260 | 130.00 |
For each endpoint, compare request_count and the token totals against how it is billed in system.billing.usage: move a high-volume endpoint off pay-per-token onto provisioned throughput, drop a low-traffic one off expensive provisioned throughput, and open a ticket on any endpoint whose error_requests ratio is high or whose last_request_date has gone stale.
Feeds into: cost_serving_cost_mode_efficiency (gov-7) — per-endpoint request COUNT in the window, to cross against billed serving mode