Skip to main content

Guide to the data

How to use the data archive

This page is a guide to what’s in the archive. It covers what the data consist of, where they came from, the choices made in assembling them, and checks on them.

Terms

  • Vintage date: the date a value was published.
  • Reference period: the period a value describes, such as 2024 Q1 or March 2024. It is a proper period, not a timestamp.
  • Change-only storage: a row is kept only when a value is first published or later revised. Any past vintage can be rebuilt from these rows.
  • CDID: the four-character ONS code for a single time series, for example ABMI for GDP.

The data

The archive consists of a single table in parquet format. Each row gives a single economic value, its reference period (the time the number describes, such as a particular quarter or month), and the vintage date (the day the number was published), plus other meta-data.

A row appears only when there is new information. If a number is republished unchanged, no row is added. You can still rebuild the figures as they stood on any date, because the most recent row on or before that date is the figure that was then in force. (ALFRED, the St. Louis Fed’s archive of US data, stores its data the same way.)

194

series

294,559

rows

1,563

distinct publication dates

Sep 1961

earliest publication

Nov 2026

latest publication

Of the rows, 20,238 are first estimates, 252,141 are revisions, and 22,180 are old figures that the archive first saw long after the period ended. Those last ones are labelled history and should not be treated as first estimates.

What the data cover

GDP and its components, output by industry, prices, the labour market, trade, and household and government finances, along with 129 results from four business surveys. Most official series come from the Office for National Statistics (ONS), the UK’s national statistics office, and have a CDID, which is ONS’s four-character code for a time series (ABMI is GDP, for example). Where ONS publishes a growth-rate table with no code of its own, the archive uses a label based on the matching code, such as IOS_MOM for monthly growth in services. Survey series have codes of our own, such as DMP_SALES_REALISED_MEAN.

A heatmap with one row per series and one column per year of publication, beside a panel of bars showing the range of periods each series covers. Only GDP reaches back to the earliest years. Some series begin decades later and many begin only in the most recent stretch. The periods described stretch back further than the publications. The four business surveys appear together as a final group and begin much later than GDP.
Figure 1: Left: how many separate publications are held for each series in each year. Right: the span of periods that each series’ figures describe. Series are grouped by subject, and each survey is shown as one row because all its series come from the same files.

How to read Figure 1:

  • Depth varies a lot between series. GDP is held back to 1961. 32 series have publications from before 2000. 28 official series start in 2015 or later. Most of these rely on the ONS website’s own page history, which begins around then, and a few only began being published later (monthly GDP dates from 2018).
  • A step up from September 2015. The band of publications thickens sharply from that point, because that is when the page histories begin.
  • Periods described reach further back than publications. A table published in 2015 can include figures for 1946. The left panel shows when the archive knows something was said, and the right panel shows what it was said about.

A single series can hold annual, quarterly and monthly figures at once, as ONS often publishes all three under one code. The frequency column separates them, so filter on it before comparing values. Rolling three-month labour market figures are marked in the reference_period_basis column and stored against the month the window ends.

Many indicators appear twice, once as a level (ABMI, GDP in pounds) and once as a growth rate (IHYQ, GDP growth on the previous quarter). The two forms come from different ONS products and are kept apart on purpose.

Where these data come from

The figures come from files published by ONS and the Bank of England, gathered by 8 routes. The source column names the publisher and the route column names the route:

  • Past versions of ONS web pages. ONS keeps every version of each series’ page. These give exact publication dates, but mostly only from September 2015. Example: GDP.
  • Revisions triangles. These are ONS spreadsheets that lay out each estimate of a period as it was revised, row by row. They often reach back well before the page histories but record only the publication month. Example: GDP revisions triangle.
  • Real-time databases. These are ONS spreadsheets holding each published version of GDP and its main expenditure components, by month. GDP goes back to 1961. Example: GDP real-time database.
  • Never-revised series. ONS does not revise some price measures, such as the Retail Prices Index, so today’s figure is also the first one published. Example: Retail Prices Index.
  • Business surveys. The Business Insights and Conditions Survey comes from ONS. The Decision Maker Panel, the Credit Conditions Survey and the Agents’ scores come from the Bank of England. The first three publish a new file for each release, so each file is a true vintage. The Agents’ scores come as a single file, so their dates are estimated from the Bank’s report dates. Example: Decision Maker Panel.
A stacked bar chart of the share of rows per publication year by source. The earliest years come entirely from real-time databases, then revisions triangles supply a growing share, and in the most recent years past versions of ONS pages supply most rows, with the four survey routes making up part of the rest.
Figure 2: Share of the rows published in each year, by the route they came from. The older routes supply the early decades, the ONS page histories take over in the most recent years, and the surveys add a further layer from their own files.

Figure 2 shows that 60% of rows come from page histories and 66% of rows are dated to the exact day. Each row records its source file, that file’s fingerprint (a hash, which changes if the contents change) and when it was downloaded. Each raw file is kept, so the table can be rebuilt without going back to the publishers.

Judgements made in assembling the data

These are the choices most likely to affect your results.

If vintage is uncertain, we err on the late side. Dates must never suggest a figure was known before it was. Older sources give only a publication month, so those rows are dated to the last day of that month. A model using the archive may therefore see an old figure a few weeks after it came out, but we aim to to prevent it from appearing before it came out in reality. The vintage_date_precision column says how each date was arrived at: day for exact, month for the end of the known month, and upper_bound for the latest date it could have been.

Dating gets sharper from September 2015. Before then, nearly all dates are month-ends. Afterwards, most are exact. This is due to how ONS makes revisions and vintages available.

One source at a time. When sources overlap, the one with the most exact dates is preferred. Page histories rank first, then triangles and never-revised series, then real-time databases.

Rounding differences are not revisions. Two sources may give the same number to different decimal places. Values filling a gap are rounded to match the series’ main source, so a change in rounding does not show up as a false revision.

Old figures are labelled as old. When a period first appears in the archive more than a year after it ended, no source reaches back to its real first estimate. Such rows are marked history. For example, GDP growth for 1955 to 1991 first appears in a table from May 2015. To study first estimates only, use the rows labelled first.

Never-revised series are filled in backwards. Before September 2015, CPI inflation and RPI come from today’s figures, each dated to the end of the month after the month it describes. That is later than the true release, but never earlier.

Numbers are kept as published. No rebasing, smoothing or correcting has been done. Levels therefore change units every few years, as ONS moves its figures to the prices of a new reference year, and the oldest GDP vintages are in the prices of much earlier years. Compare levels only within a single vintage.

Growth rates are better for testing models. Because levels change units, the published growth rates are more consistent through time. Which series to use lists a recommended series for each main indicator.

An example: UK GDP and its revisions

A line chart with one thin line per publication date, each showing UK GDP as an index. Lines spread apart as they move right, showing that the path of GDP was revised. Early vintages are violet and recent ones bright cyan, with the latest in white. Vintages differ most around 2008, and all recent ones show the sharp fall in 2020.

Figure 3: UK GDP as published at each date since 1997. Each line is one vintage: the whole history of GDP as ONS showed it on one publication date. To remove changes of units, each vintage is set to 100 in 1997 Q1. Older vintages are darker. (NB: Figure 3 does not include vintages from before 1997.)

The lines start together at 100 and spread apart as ONS gathers fuller information and improves its methods. The spread is the reason for keeping every vintage: someone looking at GDP years ago saw a different past from someone looking today.

Single quarters show the same thing. ONS first estimated growth in the fourth quarter of 2008 at −1.5% in January 2009. The archive holds 12 changes to that figure since then, and it now stands at −2.1%. Growth in the second quarter of 2020 was first put at −20.4% on 12 August 2020, has changed 9 times, and now stands at −19.9%.

This chart uses the levels series, indexed within each vintage, because levels show how the whole path shifted.

Are these data and their vintages likely to be correct?

Values could be incorrect, or publication dates, or both. In this section, we show several tests that try to verify the accuracy of the values and the vintages. (Of course, this doesn’t address other issues, like measurement error!)

Comparison with an independent archive

The OECD has kept its own record of UK statistics, one edition a month since February 1999, shown in Figure 4. These data are independent of the archive. Each comparison takes the most recent periods in each OECD edition, works out growth rates and checks them against the archive around the same date. That covers 40,107 comparisons on 13 of the archive’s 194 series, across up to 333 editions. These results were last worked out on 2 November 2026.

A scatter chart of differences between the archive and the OECD for quarterly GDP growth, by OECD edition from 1999 to 2026. Nearly all dots sit on or close to zero and inside a narrow band of plus or minus 0.05 percentage points, with only a few outliers.
Figure 4: Each dot compares one GDP growth figure in an OECD edition with the archive’s figure on the day before that edition’s month began. Dots inside the band agree to within rounding. A few fall outside because the OECD collected a release at a different time from the one this strict test assumes.

A dot plot with one row per comparison and three markers each. With the looser tolerance, every comparison sits close to complete agreement. With the tight tolerance, most sit a little lower, while gross fixed capital formation, the GDP deflator, CPI, CPIH and the production growth triangle sit noticeably lower.

Figure 5: The test uses the archive’s figures from the day before the OECD edition’s month. The nearby tests allow the closest vintage from 62 days before to 31 days after the start of that month, because the OECD may have collected figures a little earlier or later.

Under the looser test, at least 99.3% of figures agree in every comparison. The tighter tests performs less well, but there are two reasons that this may be the case. First, the OECD rebases ONS’s one-decimal indices, so growth worked out from them can sit a little away from the rate ONS published, even for the same release. That accounts for much of the gap in production and in consumer prices. Second, the OECD may have collected a release at a different point from where the test looks.

Unemployment and retail sales are not compared, because the OECD’s UK series are different measures from the ONS ones held here. Most other series have no outside check of this kind.

Comparison with itself

ONS publishes some series both as levels and as growth rates, and the archive often gets them by different routes. At each vintage, growth worked out from the levels should match the published growth rate. A mismatch would point to a misread or misdated file.

Growth and level series Figures compared Within rounding Within 0.15 points Largest gap
IHYQ and ABMI 34,265 98.2% 99.3% 0.47 points: 1998Q1 as of 1998-05-31
YBHA_QOQ and YBHA 18,990 99.9% 100.0% 0.34 points: 2004Q1 as of 2005-03-31
ABJR_QOQ and ABJR 14,589 99.8% 99.9% 0.53 points: 1998Q4 as of 1999-06-30
NPQT_QOQ and NPQT 14,744 99.7% 99.8% 5.46 points: 2002Q2 as of 2002-08-31
NMRY_QOQ and NMRY 14,844 99.9% 100.0% 1.69 points: 1997Q1 as of 1999-07-31
IKBK_QOQ and IKBK 14,714 99.7% 99.9% 1.11 points: 2021Q1 as of 2021-08-31
IKBL_QOQ and IKBL 14,816 99.7% 99.8% 0.48 points: 1998Q3 as of 1999-07-31

One of the largest gaps is the August 2002 typing error mentioned above, which sits in the published growth rate. Of the 7 largest gaps, 4 are in vintages from before 2000. The early GDP gaps remain unexplained (see the limits below).

Checks on dating

  • Figures published before their period ended. A figure dated on or before the end of the period it describes would suggest a wrong date. 24 of 294,559 rows (0.008%) meet this test. 22 are annual workforce jobs figures that appear before their year is over, published that way on ONS’s page, so they do not look like dating errors. Surveys are left out of this test, because they can properly report on a month before it ends. The other 2 are production and manufacturing figures (for example, 2021Q3 published 2021-09-10), which have not been looked into further.
  • First figures that arrive late. If a period appeared in the archive later than in one of the sources collected, a source would have been wrongly set aside. There were none when this was last checked, on 2 November 2026.
  • Mixed definitions. A monthly series should not mix single months with rolling three-month windows. 0 series do.

Traceable values

The archive is put together by a program from stored copies of the published files, with no figures entered or changed by hand.

Known limits

See Data Issues for full details of the limitations of this archive.

Which series to use

Several indicators are held in more than one form. If you are testing a model on the data available at each date, choose the form that stays the same throughout your sample. In practice, that usually means the published growth rate.

Indicator Series Publications from Notes
GDP, quarter on quarter IHYQ 1992
GDP, month on month ECYX 2018 Monthly GDP was first published in July 2018
Nominal GDP, quarter on quarter YBHA_QOQ 1992
GDP deflator, on a year ago YBGB_YOY 2006
Household consumption ABJR_QOQ 1996
Government consumption NMRY_QOQ 1996
Gross fixed capital formation NPQT_QOQ 1996 The August 2002 vintage holds an ONS typing error
Business investment KG7P 1998
Exports (volume) IKBK_QOQ 1996
Imports (volume) IKBL_QOQ 1996
Production, month on month IOP_MOM 1998
Manufacturing, month on month IOM_MOM 1998
Services, month on month IOS_MOM 2000
Construction, month on month CON_MOM 2013
Retail sales volume, month on month RSI_MOM 1998 Not checked against the OECD
CPI inflation D7G7 1997 Never revised, so earlier values are first publications
RPI CHAW 1997 Never revised
Unemployment rate MGSX 1999 Rolling three-month figures
Employment rate LF24 2010 Rolling three-month figures
Claimant count BCJD 1999
Household saving ratio NRJS 1998

What each column means

Column Meaning
series_id The publisher’s code for the series, or a label of our own
dataset_id The dataset or survey the value came from, such as QNA or DMP
title, short_name, measure What the series is and its units
frequency A annual, Q quarterly, M monthly or D daily, for survey waves
reference_period The period the value describes
reference_period_basis period, 3m_ending for rolling three-month figures, or survey
vintage_date When the value was published
vintage_date_precision day, month or upper_bound
release_type first, revision or history
value The published value
source The publisher: ONS or Bank of England
route, source_url The route and the exact file or web address
retrieved_at, content_sha256 When the file was downloaded, and its fingerprint